AWS US-EAST-1 Outage (October 2025) Explained: How a DNS Race Broke DynamoDB

Every program that calls the service by name gets no address back, so it cannot connect at all. That is how the AWS us-east-1 outage of October 2025 began. In AWS's account, the first failure was this name lookup.

A DNS record is the entry that maps a name, such as a website's address, to one or more IP addresses. An IP address is the number a computer actually connects to. If the record holds no addresses, the name leads nowhere.

Picture a common setup. An Indian bill-payments app runs its API servers on EC2 in us-east-1 and keeps user sessions and orders in DynamoDB. The app is made up, but every failure it meets below comes from AWS's own summary.

At 12:18 PM IST on Monday 20 October 2025, this app's servers start failing their database calls. Logins break, payments stall, and the support queue fills. The servers themselves are healthy, which makes the problem confusing at first.

You need about nine terms. Each one below is used later exactly as defined here, so you can come back to this list.

• Region: a group of AWS data centres in one geographic area. us-east-1 is Northern Virginia in the United States.
• Availability Zone (AZ): one or more separate data centres inside a region, with their own power and networking.
• Endpoint: the name your code calls to reach a service, such as dynamodb.us-east-1.amazonaws.com.
• Route 53: AWS's DNS service. It stores and answers the DNS records for these endpoints.
• TTL (time to live): how many seconds a computer may keep a DNS answer in its cache before asking again.
• Control plane: the parts that make changes, such as launching servers. Data plane: the parts that do the daily work, such as serving requests.
• Lease: a time-limited claim one system holds over another. If it is not renewed in time, it expires.
• Health check: a small test request that decides whether a server should receive traffic.
• Post-event summary: AWS's name for its official post-mortem, the written account of what failed and why.

An automation bug emptied the DNS record for DynamoDB's main regional endpoint. AWS's post-event summary calls it a latent race condition, which means a bug that was always there but needed rare timing to show up.

AWS describes three distinct periods of impact. DynamoDB returned errors first. EC2, AWS's virtual server service, then failed to launch new servers. Network Load Balancers (NLB), AWS's service that spreads connections across servers, then returned connection errors. Other AWS services that depend on these were hit too.

The summary names Lambda, ECS, EKS, Fargate, Amazon Connect, STS, IAM console sign-in, Redshift and the Support Center among the affected services. Some Redshift clusters were only fully restored by 4:05 AM PDT on 21 October.

From first error to the end, the event lasted 14 hours 32 minutes. The DynamoDB part lasted 2 hours 52 minutes. The rest came from systems that depended on DynamoDB and could not recover cleanly.

DynamoDB used two separate automated parts to keep its DNS records correct. The DNS Planner decided what the records should say. The DNS Enactor wrote those decisions into Route 53.

AWS says DynamoDB maintains hundreds of thousands of DNS records for a very large fleet of load balancers. It also runs variant endpoints, such as a FIPS endpoint that meets US government cryptography standards, an IPv6 endpoint and account-specific endpoints. AWS uses automation to manage them.

What the DNS Planner did

The Planner watched the health and capacity of DynamoDB's load balancers. It regularly produced a new DNS plan: a set of load balancers and weights, where a weight says how much traffic each one gets. AWS made a single regional plan shared across endpoints, which it says simplifies capacity management.

Each new plan was a new generation. Think of generation numbers as version numbers for plans: a higher number is newer.

What the DNS Enactor did

The Enactor picked up plans and applied them in Route 53. AWS says it was designed with minimal dependencies so it could help recover the system in any scenario. For resilience, three copies ran fully independently, one in each of three Availability Zones.

Each copy replaced the current plan using a Route 53 transaction. Route 53's documentation says a change batch is all or nothing, so an endpoint never ends up with half a plan. Transactions stopped two copies from mixing plans, but they did not decide which plan was newest.

A slow Enactor wrote an old plan after a fast Enactor had written a new one, and then the fast Enactor's clean-up deleted that old plan. The plan the endpoint pointed at no longer existed, so the record had no addresses.

Before applying a plan, each Enactor made a one-time check that its plan was newer than the one already applied. AWS calls it a one-time check, made at the start of applying the plan. While an Enactor worked through endpoints, it could be blocked by another Enactor and would retry.

1. 1

   One Enactor is delayed

   Just before the event, one Enactor hit unusually high delays. It kept retrying its update on several endpoints, carrying a plan that was getting older.

2. 2

   The Planner keeps going

   Meanwhile, the Planner produced many newer generations of plans. It kept running while one Enactor was slow.

3. 3

   A second Enactor finishes fast

   A second Enactor picked up one of the newest plans. It quickly applied that plan to all endpoints.

4. 4

   Clean-up starts

   Having finished, the second Enactor ran the plan clean-up process. Clean-up deletes plans that are significantly older than the plan it just applied.

5. 5

   The old plan overwrites the new one

   At the same moment, the delayed Enactor applied its much older plan to the regional endpoint. Its newer-plan check was stale by then, so nothing stopped it.

6. 6

   The active plan is deleted

   Clean-up deleted the old plan, because it was many generations behind. That plan was now the active one, so all IP addresses for the endpoint vanished.

7. 7

   Automation is stuck

   With the active plan gone, AWS says the system was in an inconsistent state. No Enactor could apply later plans, and operators had to fix it by hand.

Here is an everyday picture. A school notice board shows today's timetable. One teacher is slow to pin up yesterday's sheet. Another teacher pins today's sheet, then throws away all old sheets from the staff room. The slow teacher pins yesterday's sheet on top, and it is then thrown away. The board is now empty.

Map it back: the board is the DNS record, the sheets are plans, the teachers are Enactors and the throwing-away is clean-up. Where the picture breaks: a real board would still show the sheet underneath, while a DNS record holds only the active plan.

For our payments app, this is the moment everything breaks. Its servers ask DNS for the DynamoDB endpoint and get no address. AWS says customers could resolve the endpoint again as cached DNS answers expired between 2:25 AM and 2:40 AM PDT.

Yes, as a toy model. The code below is our simplified sketch of the sequence AWS describes, not AWS's code. It keeps plans as numbered generations and shows the record going empty.

from operator import gt

# A toy model of the race, not AWS code. Plans are numbered by generation.
plans = {1: ["10.0.0.1"], 2: ["10.0.0.2"], 3: ["10.0.0.3"], 4: ["10.0.0.4"], 5: ["10.0.0.5"]}
dns = {"active": 1}          # which plan the endpoint points at
KEEP = 2                     # clean-up deletes plans more than this many generations older

def records():
    return plans.get(dns["active"], [])

def check_is_newer(gen):     # the one-time check, done only at the start
    return gt(gen, dns["active"])

def apply(gen):
    dns["active"] = gen

def clean_up(just_applied):
    for gen in list(plans):
        if gt(just_applied - gen, KEEP):
            del plans[gen]

# Enactor A picks up plan 2 and passes its check, then gets stuck retrying.
a_plan = 2
a_ok = check_is_newer(a_plan)
# Meanwhile the planner makes newer plans and Enactor B applies plan 5.
b_plan = 5
if check_is_newer(b_plan):
    apply(b_plan)
print("after B applies:", dns, records())
# A finally writes its old plan; its check result from the start is stale.
if a_ok:
    apply(a_plan)
print("after A overwrites:", dns, records())
# B's clean-up now deletes plan 2, which is the active one.
clean_up(b_plan)
print("after clean-up:", dns, "records =", records())

# The guarded version: check again at write time, never delete the active plan.
plans = {1: ["10.0.0.1"], 2: ["10.0.0.2"], 3: ["10.0.0.3"], 4: ["10.0.0.4"], 5: ["10.0.0.5"]}
dns = {"active": 1}
def apply_guarded(gen):
    if check_is_newer(gen):  # compare-and-set at the moment of writing
        dns["active"] = gen
        return True
    return False
def clean_up_guarded(just_applied):
    for gen in list(plans):
        if gt(just_applied - gen, KEEP) and gen != dns["active"]:
            del plans[gen]
apply_guarded(5)
print("guarded A write accepted?", apply_guarded(2))
clean_up_guarded(5)
print("guarded result:", dns, "records =", records())

Run it with python3 race_sim.py. We ran it on Python 3.13.17 and got this output, line by line:

after B applies: {'active': 5} ['10.0.0.5']
after A overwrites: {'active': 2} ['10.0.0.2']
after clean-up: {'active': 2} records = []
guarded A write accepted? False
guarded result: {'active': 5} records = ['10.0.0.5']

The third line is the outage in miniature: the endpoint points at plan 2, and plan 2 has no records. The guarded version rejects the old write because it checks again at the moment of writing. Its clean-up also refuses to delete whatever is active.

Because EC2's internal server manager depends on DynamoDB, and it fell so far behind that catching up overwhelmed it. DynamoDB's DNS was restored at 2:25 AM PDT, but new EC2 launches kept failing until 10:36 AM PDT, and EC2 was not fully recovered until 1:50 PM PDT.

Existing EC2 instances that were already running stayed healthy throughout, according to AWS. Only changes failed: launching new instances and the network setup for them. For our payments app, the running servers kept going, but autoscaling, which adds servers as load grows, could not add more.

Leases between DWFM and droplets

AWS calls the physical servers that host EC2 instances droplets. A subsystem called DropletWorkflow Manager, or DWFM, manages them. DWFM holds a lease on each droplet, which lets it track the droplet's state and apply EC2 actions correctly.

To keep a lease, each DWFM host must regularly check in and complete a state check with each droplet. Those state checks depend on DynamoDB. From 11:48 PM PDT they began to fail, and leases started timing out across the fleet.

Congestive collapse during recovery

When DynamoDB returned at 2:25 AM PDT, DWFM began renewing leases with droplets across the whole EC2 fleet. There were so many that renewals took too long and timed out. Timed-out work was queued again, adding to the pile.

AWS calls this state congestive collapse: the system was busy but made no forward progress. A droplet without a lease cannot receive new launches, so customers saw insufficient capacity errors. AWS says there was no established recovery procedure for this situation.

At 4:14 AM PDT, engineers throttled incoming work and restarted DWFM hosts selectively. Restarts cleared the queues and cut processing times. By 5:28 AM PDT, DWFM held leases with every droplet in the region.

The Network Manager backlog

A second subsystem, Network Manager, sends network configuration to new instances so they can talk to the network. Its work had been delayed too, so a large backlog was waiting. At 6:21 AM PDT its propagation times grew long.

New instances could launch but had no working network connection until their configuration arrived. Engineers reduced the load on Network Manager, and propagation was back to normal by 10:36 AM PDT. Request throttles were then relaxed from 11:23 AM, with full EC2 recovery at 1:50 PM.

The health checker brought new EC2 instances into service before their network configuration had fully arrived. Healthy nodes failed checks, got removed from DNS, then passed and came back. That constant flapping removed real capacity.

AWS says the alternating results also overloaded the health check subsystem itself. It slowed down, and the delays triggered automatic AZ DNS failover. That feature moves traffic away from an Availability Zone that looks unhealthy.

For load balancers spread over several AZs, failover took capacity out of service. Apps saw connection errors when the remaining capacity could not carry their load. Monitoring caught it at 6:52 AM PDT.

At 9:36 AM PDT, engineers switched off automatic health-check failover for NLB. That brought every healthy node and target back into service and ended the connection errors. AWS switched failover back on at 2:09 PM PDT, after EC2 had recovered. AWS still counts the NLB impact window as running to 2:09 PM, and the summary does not say what impact remained after 9:36 AM.

AWS listed four concrete changes in its post-event summary, plus a general promise. The first was already done when the summary was written.

• DynamoDB: the DNS Planner and DNS Enactor automation was disabled worldwide. AWS says it will fix the race and add protections against incorrect DNS plans before turning it back on.
• Network Load Balancer: a velocity control, which is a speed limit on change, will cap how much capacity one NLB can remove when health checks trigger AZ failover.
• EC2: an extra test suite, on top of existing scale testing, will exercise the DWFM recovery workflow to catch future regressions.
• EC2: the throttling in its data propagation systems will rate-limit incoming work based on the size of the waiting queue.
• All services: AWS says it will look for more ways to avoid similar impact and to reduce recovery time.

As of 23 October 2026, the summary page gives no date for re-enabling the DNS automation. Check the page again before quoting the status.

Six patterns from this event apply to ordinary systems too. These are our readings of the summary, checked against AWS's own published design guidance, not lessons AWS itself listed.

Check again at write time, and never clean up what is live

The Enactor checked that its plan was newer once, at the start, and then spent a long time retrying. A check-then-act gap like this is the classic shape of a race condition. Put the check inside the same atomic write, often called compare-and-set, so a stale decision cannot land. Compare-and-set means the write succeeds only if the current value is still the one you checked.

The clean-up job deleted plans by age. An old plan can still be the active one if something wrote it late. A safe clean-up asks what is live right now and skips it, however old it looks.

Keep the data plane working without the control plane

AWS's Builders' Library defines a statically stable design as one where the system keeps working even when a dependency becomes impaired. Running EC2 instances did exactly that. Our payments app could do the same for reads by caching sessions, or by failing over to a global-table replica.

Slow down retries and automatic removal

DWFM's timed-out renewals were queued again, which added load to an overloaded system. AWS's guidance on retries says to back off exponentially with jitter, meaning growing, randomised waits, and to limit retries. Its example: five layers that each try a call three times can multiply database load 243 times.

The NLB flapping is the same lesson for automation. A health checker that can remove any amount of capacity can remove healthy capacity too. AWS said it is adding a velocity control to cap how much one NLB can remove during AZ failover.

Shrink the blast radius with cells

Blast radius is the share of users one failure can reach. One endpoint name served every caller in the region that used it, so one bad write reached all of them. AWS's cell-based architecture paper describes cells as isolated copies of a workload that share no state. With 10 cells, one failing cell leaves 90 percent of requests untouched.

Use it as a story about hidden dependencies and recovery. Interviewers care less about the exact times and more about whether you can see the chain and design against it.

1. 1

   State the trigger in one sentence

   Two copies of DynamoDB's DNS automation raced. A stale plan overwrote a fresh one, clean-up deleted it, and the regional endpoint resolved to nothing.

2. 2

   Name the race precisely

   Say it was a check-then-act gap: a one-time newer-plan check made at the start went stale during long retries.

3. 3

   Trace the dependency chain

   DynamoDB failed, so EC2's lease manager lost its leases. Mass renewal caused congestive collapse, then a network backlog, then flapping NLB health checks.

4. 4

   Separate control plane from data plane

   Point out that running instances survived while launches failed. That is static stability working, and it shows where to invest.

5. 5

   Propose concrete fixes

   Offer compare-and-set writes, clean-up that skips live items, backoff with jitter, throttles based on queue size and capped automatic capacity removal.

6. 6

   Close with blast radius

   Explain how cells, isolated copies of a service, would limit one bad write to part of the traffic.