The 2017 AWS S3 Outage Explained: One Typo, Four Hours

Everything that depends on those servers stops, all at once, and the person who typed the command often finds out last. This guide uses the AWS S3 outage 2017 report to show how that happens, and how to prevent it.

Picture a common setup. An online shop prints designs on cushion covers, bags and dupattas, and keeps its 42,000 product images in cloud object storage in one region. The setup is made up, but the failure pattern is real.

Between the storage and the website sit 12 image servers. They fetch each photo, resize it for phones or laptops, and send it on. On a sale morning, the one engineer who runs the shop's servers starts a script to retire 2 idle image servers.

He types 12 instead of 2. The script obeys without a question and removes every image server. For 50 minutes, every product page shows grey boxes where the photos should be.

About 140 orders are lost, roughly Rs 1 lakh (100,000 rupees) of sales. Nobody buys a cushion cover they cannot see. The founder asks how one wrong number could take down the whole shop.

On 28 February 2017 the same kind of mistake disrupted Amazon S3 in its northern Virginia region for over four hours. AWS published a post-event summary afterwards. Every fact about the outage below comes from that summary.

You need five words: object storage, region, subsystem, playbook and blast radius. Each one gets a plain meaning and an example from the shop.

Object storage. Object storage keeps files, called objects, under a name called a key, inside a container called a bucket. Amazon S3 (Simple Storage Service) is the AWS object storage service. Example: the shop keeps a cushion photo under the key cushions/peacock-red.jpg.

Region. A region is a group of data centres in one geographic area, which a cloud provider runs separately from its other regions. The 2017 outage hit US-EAST-1, the northern Virginia region. Example: all of the shop's images sit in one region.

Subsystem. A subsystem is one part of a big service, with its own job and its own servers. The AWS report mentions three S3 subsystems: one used by the billing process, plus index and placement. Example: the shop's image servers are one subsystem, and checkout is another.

Playbook. A playbook is a written, step-by-step procedure for a routine task, like retiring servers. Engineers follow it so the task is done the same way every time. Example: the shop's playbook for retiring servers is one wiki page plus one script.

Blast radius. Blast radius is how much of a system one failure can damage. A small blast radius means a fault hurts a few users; a large one means it hurts everyone. Example: the shop's typo had a blast radius of the whole shop.

One wrong input to a routine command caused it. The command removed a larger set of servers than intended, and some of those servers supported two core S3 subsystems.

The S3 team was debugging an issue that made the S3 billing system run more slowly than expected. At 9:37 AM Pacific time, an authorized team member used an established playbook to remove a small number of servers from one S3 subsystem.

One input to the command was entered incorrectly. The servers removed by mistake supported two other subsystems, called index and placement. The report says removing a significant portion of their capacity forced each of them into a full restart.

Capacity here means how much work a subsystem can handle, roughly how many healthy servers it has. While index and placement restarted, S3 in the region could not serve requests.

The summary does not say how many servers were removed, which input was wrong, or how big the index is. This guide does not guess those numbers.

What the index and placement subsystems do

The index subsystem manages the metadata and location of every S3 object in the region. Metadata means data about data, such as an object's name, size and where its bytes sit.

The report says the index is needed to serve all GET, LIST, PUT and DELETE requests. Those are the basic S3 requests: GET reads an object, PUT writes one, LIST shows what is in a bucket, and DELETE removes one.

The placement subsystem allocates storage for new objects, so it is used during PUT requests. The report says placement needs the index working properly to operate correctly.

The diagram below is a simplified picture, because AWS has not published S3's internal design beyond the report. Follow a request from the shop's product page to the stored bytes, and notice that every path needs the index.

Why one wrong number could do so much

The difference was speed and the lack of a floor. AWS's own verdict was that "the tool used allowed too much capacity to be removed too quickly." Nothing stopped the command before a subsystem fell below what it needed.

Picture a dam with 30 sluice gates, the sliding doors that let water out of the reservoir. An operator means to open 2 gates. He enters 20 by mistake, and the control panel simply obeys.

Here is how the picture maps. The water is the subsystem's capacity, the turbines are the requests it serves, and the control panel is the removal tool. A safe panel would refuse any order that drains the lake below a marked level.

Where the picture breaks: water drains away smoothly, but S3 capacity has a cliff. Below some unpublished level, index and placement could not keep running and had to restart from zero.

S3 in US-EAST-1 was disrupted from 9:37 AM to 1:54 PM Pacific time on 28 February 2017, which is 4 hours 17 minutes. In India that was from 11:07 PM to 3:24 AM the next morning.

Notice the gap: 2 hours 49 minutes passed before reads came back, and writes needed another 88 minutes. The next two sections explain why the damage spread so wide and why recovery took so long.

The outage spread because other AWS services in US-EAST-1 rely on S3 for storage, so they failed while S3 could not serve requests. The report names the S3 console, new EC2 instance launches, some EBS volumes and AWS Lambda.

EC2 is the AWS service for renting virtual servers, and EBS gives those servers disks. Only EBS volumes that needed data from an S3 snapshot, a saved copy of a disk, were hit. Lambda runs small pieces of code on demand.

This is a dependency chain: service A needs B, and B needs C. When something low in the chain breaks, everything above it breaks too. S3 sits low in that region's chain, so its blast radius was large.

Customer apps were in the same position. An app that loaded its files from S3 in that region could not fetch them, and showed broken pages like the shop's grey boxes.

The diagram shows how one command reached far beyond the billing team. Start at the command box and follow the arrows outward.

The status page that could not report its own outage

The AWS Service Health Dashboard tells customers which services are healthy. Its administration console, the screen staff use to update it, depended on S3.

So until 11:37 AM, two hours in, AWS could not update individual services' status on its own dashboard. It posted updates on its @AWSCloud Twitter feed and in banner text at the top of the dashboard instead.

The shop has the same trap. Its delivery-delay banner is a small file in the same bucket as the product images, so if that bucket fails, the delay notice fails with it.

S3 took hours because index and placement had to restart from scratch, and AWS had not completely restarted either of them in its larger regions for many years. S3 had grown massively since then.

A restart is not just switching servers back on. The report says the restart included running the safety checks that validate the integrity of the metadata, and that this took longer than expected.

The report does not say what the checks look for. The reason for them is plain, though: the index is the map of every object, and a damaged map could send requests to the wrong data.

Order mattered too. The index was the first to restart, and placement could only begin recovery once the index was working. That is why reads came back at 12:26 PM and new writes only at 1:54 PM.

Back at the dam, draining the lake was quick. Refilling it takes hours, and nobody there has restarted from an empty lake in years, so nobody knows how long it takes today.

The diagram shows the recovery in order. Follow the arrows from the first restart to the backlogs, and notice where the path splits.

AWS's fixes target the tool, not the person. The summary, written in 2017, lists five changes. Two make the removal tool safer, and the rest cover an audit of other tools, faster recovery through cells and a status dashboard spread across regions.

• Slower removal. The tool was modified to remove capacity more slowly.
• A floor. Safeguards now stop a removal that would take any subsystem below its minimum required capacity level.
• Audits. AWS said it was auditing its other operational tools for similar safety checks.
• Cells. AWS had planned further partitioning of the index into smaller cells later in 2017, and moved that work to begin immediately.
• The dashboard. Its administration console was changed to run across multiple AWS regions.

This guide reports what the 2017 summary says. It does not track how each change has evolved inside AWS since then.

Give it the same two guards AWS added: a limit on how much one run can remove, and a floor it will never cross. Then add a check AWS's summary does not mention: reject inputs that make no sense.

The sample below runs as it is on Python 3.13. The helpers drain, take_out_of_service and wait_until_healthy are stand-ins that only print. To drain a server means to stop sending it new work and let it finish what it has.

MIN_SERVERS = {"image-servers": 8, "checkout": 4}
MAX_REMOVE_PER_RUN = 2

def drain(server): print("draining", server)          # stand-in
def take_out_of_service(server): print("removed", server)  # stand-in
def wait_until_healthy(subsystem): print(subsystem, "healthy")  # stand-in

def retire_servers(subsystem, count, live_servers):
    if count < 1 or count > MAX_REMOVE_PER_RUN:
        raise ValueError(f"Refused: count must be 1 to {MAX_REMOVE_PER_RUN}, got {count}")
    left = len(live_servers) - count
    if left < MIN_SERVERS[subsystem]:
        raise ValueError(f"Refused: {subsystem} would drop to {left} servers")
    for server in live_servers[:count]:
        drain(server)                  # stop new work, finish current work
        take_out_of_service(server)
        wait_until_healthy(subsystem)  # pause before the next one

servers = [f"img-{n:02d}" for n in range(1, 13)]   # 12 live servers
for subsystem, count, live in [("image-servers", 12, servers),
                               ("image-servers", -1, servers),
                               ("image-servers", 2, servers[:9]),
                               ("image-servers", 1, servers)]:
    try:
        retire_servers(subsystem, count, live)
    except ValueError as err:
        print(err)

Run it and you should see exactly this output:

Refused: count must be 1 to 2, got 12
Refused: count must be 1 to 2, got -1
Refused: image-servers would drop to 7 servers
draining img-01
removed img-01
image-servers healthy

What just happened? The typo of 12 was refused, so was -1, and a request that would leave 7 servers hit the floor of 8. Only the safe removal of 1 server went through, with a health wait after it.

The -1 case is the one people miss. Without the count < 1 check, -1 passes the cap and the floor, because 12 minus -1 is 13. Then live_servers[:-1] removes 11 of the 12 servers.

We ran that unguarded version too: it printed draining for 11 servers. A negative number is a realistic typo, so validate the input's range, not just its size.

Two limits of this sample are worth naming. It counts listed servers, so a real tool should count healthy ones. And the floor numbers, 8 and 4, are made up; set yours from load tests.

A cell-based architecture splits one service into many isolated copies, called cells, each handling only part of the requests. When one cell fails, only its share of users is hurt.

In AWS's guidance, each cell is independent and does not share state with other cells. A thin cell router sends each request to one cell using a partition key, such as a customer ID.

The guidance gives a simple example. If 10 cells serve 100 requests and one cell fails, 90 percent of requests are unaffected.

The 2017 summary adds a second reason: recovery. AWS wrote that cells let teams assess and thoroughly test the recovery of even the largest service. A smaller unit is one you can restart in a rehearsal.

The shop applies the idea at its size. It splits the 12 image servers into 3 groups of 4: cushion covers, bags and dupattas. A bad change to one group now blanks one product family, not the whole shop.

Cells have a cost. AWS's guidance calls the router the thinnest possible layer, whose only job is routing. You also need a control plane, which creates and removes cells and moves customers between them.

Keep a copy of your important data in a second region and teach your app to read from it when the first fails. How far you go depends on what an hour of downtime costs you.

The shop's 50-minute outage cost about Rs 1 lakh, so a four-hour outage on a sale day could cost several times that. That number decides how much protection is worth paying for.

S3 can copy objects to a bucket in another region automatically. This is called Cross-Region Replication, and the copy is asynchronous, which means it happens after each write, not at the same instant. It needs versioning, which keeps every version of an object, turned on in both buckets.

S3's documentation says live replication does not copy objects that existed before you set it up. Existing objects need S3 Batch Replication, so the shop runs that once for its 42,000 images.

Then the shop changes how it loads images. The sample below runs as it is: fetch_object and StorageError are stand-ins for a real storage client, and DOWN pretends a region is out.

PRIMARY = "shop-images"           # region 1, gets all uploads
BACKUP = "shop-images-copy"       # region 2, filled by replication
PLACEHOLDER = "static/coming-soon.jpg"
DOWN = {PRIMARY}                  # pretend region 1 is out

class StorageError(Exception):    # stand-in for your client's errors
    pass

def fetch_object(bucket, key, timeout):
    if bucket in DOWN:
        raise StorageError(f"{bucket} did not answer in {timeout} s")
    return f"{bucket}/{key}"

def log_failure(bucket, key, err):
    print("FAILED", bucket, key, "|", err)

def load_product_image(key):
    for bucket in (PRIMARY, BACKUP):
        try:
            return fetch_object(bucket, key, timeout=2)
        except StorageError as err:
            log_failure(bucket, key, err)
    return PLACEHOLDER

print(load_product_image("cushions/peacock-red.jpg"))
DOWN.add(BACKUP)                  # now both regions are out
print(load_product_image("cushions/peacock-red.jpg"))

Run it and you should see exactly this output:

FAILED shop-images cushions/peacock-red.jpg | shop-images did not answer in 2 s
shop-images-copy/cushions/peacock-red.jpg
FAILED shop-images cushions/peacock-red.jpg | shop-images did not answer in 2 s
FAILED shop-images-copy cushions/peacock-red.jpg | shop-images-copy did not answer in 2 s
static/coming-soon.jpg

What just happened? With region 1 down, the image came from the copy. With both down, the page got the placeholder instead of breaking, and every failure was logged so alerts can fire.

The timeout trap in a real client

A real client does not fail as neatly as the stand-in. In October 2026 we tested boto3 and botocore 1.43.117, the AWS SDK for Python and the library under it.

We pointed the client at a local server that accepts connections but never answers, which is how a struggling service can look. No AWS traffic was involved.

The default timeouts are 60 seconds to connect and 60 to read. With both set to 2 seconds, one failed GET still made 5 attempts and took 12.7 to 19.3 seconds over three runs.

The extra time comes from retries. With no config file or AWS_RETRY_MODE setting, this version uses the legacy retry mode, which allows 5 attempts with random waits between them. A 2-second timeout is not a 2-second failure.

The retries setting is easy to misread. In the client config, max_attempts set to 1 still made 2 attempts, because there it counts retries. Setting total_max_attempts to 1 made exactly 1 attempt.

The errors are a trap too. A timeout raises ReadTimeoutError or ConnectTimeoutError, and a refused connection raises EndpointConnectionError. None of them is a subclass of ClientError, the error for answers from S3.

So a fallback that catches only ClientError crashes on the very failure it was built for. Catch the timeout and connection errors as well, or their shared parent, BotoCoreError.

This setup allows at most 2 tries per read, each with a 2-second connect timeout and a 2-second read timeout. Neither is a deadline for the whole try, and a short wait sits between tries. Run it, with boto3 installed, to check what your client will really do:

import boto3
from botocore.config import Config

cfg = Config(connect_timeout=2, read_timeout=2,
             retries={"total_max_attempts": 2, "mode": "standard"})
s3 = boto3.client("s3", region_name="us-east-1", config=cfg)
print(s3.meta.config.retries)
print(s3.meta.config.connect_timeout, s3.meta.config.read_timeout)

You should see exactly this output, where the last line is the connect and read timeouts in seconds:

{'total_max_attempts': 2, 'mode': 'standard'}
2 2

Against the silent server, this setup gave up after 2 attempts in 4.1 to 5.0 seconds over three runs, so the fallback could try the second region quickly. Defaults can change between releases. The 1.43.117 source already holds an internal switch, marked not for outside use, that makes standard mode the default when it is turned on. That is undocumented as of October 2026, so pin your version and rerun the check.

Reads are the easy half. New uploads during an outage are harder, because writes sent to the copy could clash with the primary later. The shop simply pauses new design uploads until the primary returns.

Here is the shop's new setup. A CDN (content delivery network) keeps copies of popular files on servers close to shoppers. Start at the shopper and notice that the app has three ways to show an image.

Your usual options, from least to most work:

• Cache popular files in a CDN. It is cheap and covers files already in its cache, but not new or rarely viewed ones.
• Replicate to a second region and fall back on reads. You pay to store a second copy, but product pages keep working.
• Run the whole app in two regions. It can survive losing a region, but costs much more and is hard to test well.
• Accept the risk and plan the message. For a tiny shop, a status page hosted elsewhere and a clear apology may be enough.

Choosing between these is disaster recovery planning: deciding how long you can afford to be down and how much recent data you can afford to lose.

Tell it as a short story with a clear root cause, then turn each lesson into a design choice. These six steps fit in about three minutes.

1. 1

   Tell the story in one line

   A mistyped input in a capacity-removal command took two core S3 subsystems in US-EAST-1 below what they needed, forcing full restarts.

2. 2

   Split the trigger from the root cause

   The trigger was a typo. The root cause was a tool that removed too much capacity too quickly, plus subsystems nobody had fully restarted at that size.

3. 3

   Explain the spread

   S3 sat low in the dependency chain. The S3 console, new EC2 launches, EBS volumes needing snapshot data, Lambda and the status console all needed it.

4. 4

   Explain the slow recovery

   Restarts and metadata safety checks took longer than expected after years of growth. Placement had to wait for a working index, so writes came back last.

5. 5

   List the fixes

   Slower removal, a minimum-capacity floor, audits of other tools, cells for faster recovery, and a status console that runs across regions.

6. 6

   Apply it to your own design

   Point to the admin tool in your design that can remove capacity or delete data. Say what stops a typo there, and how you would limit the blast radius.