The 2020 AWS Kinesis Outage Explained: Threads and Lessons

Everything that leans on it fails too, and the alarms often point away from the real cause. That is what many teams saw during the AWS Kinesis outage on 25 November 2020.

Picture a common setup. An analytics team in India streams clickstream events into Amazon Kinesis in the US-EAST-1 region. Clickstream means the trail of taps, page views and clicks people make inside an app.

Apps send events to the team's API servers, the programs that answer the app's requests. These servers write the events into Kinesis. Small functions read the events back and update live charts for customers. The setup is made up, but the outage and its effects are real.

At 6:45 PM India time, which was 5:15 AM Pacific time, the first alarms fired inside AWS. Here is what a team like this one would have seen, based on the effects AWS lists in its public summary.

• Writes to Kinesis returned errors. API servers that wait on each write, or retry it, slow down too.
• Alarms in Amazon CloudWatch, the AWS monitoring service, moved to INSUFFICIENT_DATA. That state means no fresh numbers are arriving, so the alarm cannot say OK or ALARM.
• Sign-ins that use Amazon Cognito, the AWS sign-in service, failed or slowed for external users.
• AWS Lambda, which runs small functions on demand, had more failed function calls. The cause was inside Lambda's hosts, not in anyone's function code.
• Updates on the AWS public status page came late, so it was hard to tell whose fault it was.

The fault was not the team's. It sat deep inside Kinesis, and it began with a change meant to give Kinesis more room.

You need six words: stream, shard, producer, consumer, partition key and shard map. The first five come from the Kinesis documentation; the last comes from the AWS report on the outage.

Stream and shard. A Kinesis data stream is a set of shards. A shard is one lane of records with fixed capacity. As of 21 October 2026, the docs give each shard up to 1,000 records and 1 MB per second of writes.

Producer and consumer. A producer puts records into the stream, like the team's API servers. A consumer gets records out and processes them, like the functions behind the charts.

Partition key. Each record carries a partition key, a short label such as a user id. Kinesis hashes the key to pick a shard, so records with the same key land in the same shard while the stream's shard layout stays the same.

Front-end and back-end. The AWS report says Kinesis has many back-end clusters that process the streams; AWS calls them cell-clusters, and cells are explained later. In front sits a front-end fleet of many thousands of servers. It handles authentication (checking who is calling), throttling (request rate limits) and routing to the right shard.

Shard map. To route a request, each front-end server keeps a shard map in memory. It is a cache, a fast local copy of data, that says which back-end cluster holds which shard.

The report says each server builds that map from three places. A small service lists the members of the fleet. Configuration comes from DynamoDB, the AWS key-value database. And the other front-end servers keep sending it messages.

The AWS Kinesis outage was a long failure of Kinesis Data Streams in US-EAST-1 on 25 November 2020. Calls to put and get records failed for most of the day, and other AWS services failed with it.

US-EAST-1 is the AWS region in northern Virginia, in the United States. A region is a group of data centres in one area.

AWS published a detailed summary afterwards. The table lists its main moments in Pacific time, as the report gives them, with India time beside them.

Look at the gaps. About 1 hour 28 minutes passed between the end of the change and the first alarm, which makes the link easy to miss. Then 4 hours 24 minutes passed before the cause was confirmed.

From first alarm to full Kinesis recovery took 17 hours 8 minutes. The next sections explain why each stretch took so long.

The root cause was that every front-end server went past the maximum number of threads allowed by an operating system setting. The small capacity addition pushed them over, and then they could not finish building their shard maps.

A thread is one line of work inside a running program. A server can run thousands at once, but the operating system caps how many can exist. The AWS report does not name the operating system or the exact limit that was hit.

The report names the design detail that mattered. Each front-end server created operating system threads for each of the other servers in the front-end fleet. The report ties them to the messages each server keeps processing from its peers. It adds that the total threads each server must keep are directly proportional to the fleet size.

Every one. With N servers, each keeps threads for N minus 1 peers. Add servers, and each existing server needs more threads too. The report says the new capacity pushed all of the servers in the fleet over the limit.

Here is an everyday picture. Each server is a desk with a shelf of fixed size. Each thread is a notebook the desk keeps for one colleague. Hire three people, and every desk needs three more notebooks.

If the shelves were nearly full, every shelf overflows on the same day, not just the new hires' shelves. That is why a small addition was enough.

Where the picture breaks: a new hire is known at once, but Kinesis servers learned about new members slowly. The report says it takes up to an hour for an existing server to learn of new participants.

The report states this right before the 5:15 AM alarms but does not link the two. Our reading: existing servers kept discovering new peers, and adding threads, after the change finished at 3:47 AM.

Once over the limit, servers could not create the threads they needed. The report says cache construction failed to complete, leaving servers with useless shard maps. A server with a useless map cannot route requests, so the requests fail.

Here is the same arithmetic as a tiny Python model. AWS did not publish its fleet size, its limit or how many threads each peer needs. The model assumes one per peer, and every number is made up.

THREAD_LIMIT = 10_000   # OS thread limit on each server (made up)
BASE_THREADS = 1_200    # threads for requests, timers and logs (made up)

def threads_needed(fleet_size):
    peer_threads = fleet_size - 1        # one thread for every other server
    return BASE_THREADS + peer_threads

for fleet_size in (8_000, 8_700, 8_900):
    need = threads_needed(fleet_size)
    status = "OK" if need <= THREAD_LIMIT else "OVER LIMIT"
    print(fleet_size, need, status)

What just happened? Run it with Python 3, and it prints three lines: 8000 9199 OK, then 8700 9899 OK, then 8900 10099 OVER LIMIT.

The line that matters is peer_threads = fleet_size - 1. It ties each server's cost to the size of the whole fleet. Growing from 8,700 to 8,900 servers, about 2 percent more, tips every server over at once.

This shape is often called all-to-all, or full mesh: every member keeps a link to every other member. Total links grow with the square of the fleet size. Ten servers need 90 one-way links; 1,000 servers need 999,000.

It took long because finding the cause was slow, and every likely fix needed a careful restart of many thousands of servers. The report says servers could only be added back at a few hundred per hour.

Finding the cause

The errors did not point at threads. The report says some errors were unrelated to the new capacity. As a precaution, the team began removing the new capacity while it kept looking.

By 7:51 AM the team had narrowed it to a couple of candidates. The leading one seemed to be creating memory pressure, meaning servers short of memory. The report says a wrong bet would have doubled the recovery time.

At 9:39 AM the team confirmed the thread limit, not memory, as the cause. It chose not to raise the operating system limit without further testing. Removing the added capacity meant the thread count would no longer exceed the limit.

Restarting a fleet that is also serving traffic

A cold start is when a server begins with an empty cache and must fill it before it is useful. Each restarted front-end server had to rebuild its shard map, using the same resources that serve incoming requests.

Bring servers back too fast, and they spend those resources building maps instead of answering. The report says such slow servers could be judged unhealthy and removed from the fleet, which sets recovery back.

1. 1

   Remove the new capacity

   Started as a precaution while the team investigated other errors. It also brought the thread count back under the limit.

2. 2

   Confirm the cause

   At 9:39 AM the team confirmed the thread limit and set aside the memory-pressure theory.

3. 3

   Change how servers start

   To speed restarts, the team added a configuration so servers got data from the authoritative metadata store (the main stored copy), not from neighbouring front-end servers.

4. 4

   Restart in groups

   The first group took Kinesis traffic at 10:07 AM. More followed at a few hundred servers per hour.

5. 5

   Watch errors fall

   The error rate dropped steadily from noon, and Kinesis was fully back to normal at 10:23 PM.

Notice step 3. During recovery, restarting servers stopped leaning on their peers for data, which the report says was done to speed restarts. A related idea returns in AWS's fixes: a dedicated cache fleet.

They broke because they use Kinesis themselves, some directly and some through another service that does. When Kinesis failed, the failure travelled along those links.

A dependency is any service your service needs to do its job. A cascading failure is when one part fails and drags down the parts that depend on it.

• Cognito sent data about API access patterns to Kinesis on a best-effort basis, through a local buffer. A latent (hidden until triggered) bug made its webservers block, that is, stop and wait, on the backlogged buffer, so sign-ins failed or slowed.
• CloudWatch uses Kinesis to process metric and log data. Calls to send metrics and logs failed or slowed, alarms went to INSUFFICIENT_DATA, and some metrics were left with gaps.
• Lambda's metric agents buffer metric data locally for a while. That buffer grew until it caused memory contention on the hosts that run functions, and errors rose.
• Reactive Auto Scaling policies, which add or remove servers based on CloudWatch metrics, were delayed until the metrics began to recover at 5:47 PM.
• CloudWatch Events and EventBridge, which pass events between programs, had API errors and delayed events from 5:15 AM.
• ECS and EKS, the AWS services for running containers (apps packed with everything they need to run), use EventBridge for internal workflows. New clusters, scaling and task clean-up were delayed.

Best-effort means nice to have: if it fails, the main job should carry on. Cognito's usage stream was designed that way, but the bug turned it into a blocker. Lambda's metric buffer hurt its hosts in a similar way.

The status page story is the sharpest lesson. The tool AWS used to post to its public Service Health Dashboard used Cognito. A backup tool existed, but it was more manual and less familiar to support operators, so updates were delayed.

On Linux, you can read the machine-wide caps, count one process's threads and count all threads in a few commands. Tracking these numbers on a chart is how you see a limit coming before it hits.

Linux has more than one ceiling. The kernel's threads-max setting caps threads for the whole machine. Every thread also takes a process id, so the kernel's pid_max setting caps them too.

A per-user cap shows as ulimit -u. Containers can add another: the pids limit of a cgroup, the Linux feature that groups processes and caps what they use. Memory limits inside one process can stop it even sooner.

1. 1

   Read the machine-wide caps

   Run cat /proc/sys/kernel/threads-max and then cat /proc/sys/kernel/pid_max. On our 8 GB test machine they printed 64086 and 32768. The smaller number is the tighter cap.

2. 2

   Count one process's threads

   Run grep Threads /proc/PID/status, with PID replaced by your server's process id. A Python process with 20 extra threads printed Threads: 21.

3. 3

   Get the same count with ps

   Run ps -o nlwp= -p PID. NLWP means number of lightweight processes, which on Linux are threads. It printed 21 for the same process.

4. 4

   Count every thread on the machine

   Run ps -eLf --no-headers | wc -l. It prints one line per thread, so the count is the total. Ours printed 135.

5. 5

   Check the per-user cap

   Run ulimit -u in the shell that starts your server. Ours printed 32043. In our test, root ignored this cap and a normal user stopped at it.

These outputs come from Linux kernel 6.18 with procps-ng 4.0.4, run on 21 October 2026. Your numbers will differ. The kernel docs say threads-max is set at boot so thread structures use at most one-eighth of RAM.

Neither machine-wide cap was the first to bind in our test. One Python 3.13.16 process, run as root with no cgroup limit, stopped at about 22,300 threads. By then it had reached the per-process memory-map limit, 65530 in /proc/sys/vm/max_map_count.

Watch a thread-per-peer design hit a limit

The demo below starts one waiting thread per imaginary peer until the system refuses, or until a safety cap of 5,000. It never touches the network. Save it as peers.py.

import threading

MAX_PEERS = 5_000  # safety cap, so the demo stays small without a limit
stop = threading.Event()
started = 0
try:
    while started < MAX_PEERS:
        # one waiting thread per peer, like a front-end server
        threading.Thread(target=stop.wait, daemon=True).start()
        started += 1
    print("threads started:", started, "(no limit hit)")
except RuntimeError as err:
    print("threads started:", started)
    print("error:", err)
stop.set()

Run plainly, it usually prints threads started: 5000 (no limit hit). To see the failure, give it a small limit, for example: docker run --rm --pids-limit 100 -v "$PWD":/app python:3.13-slim python /app/peers.py.

What just happened? Our sandbox had no Docker daemon, so we set the same kind of Linux pids limit, 100, in the kernel's cgroup files. On Python 3.13.16 it printed threads started: 99, then error: can't start new thread.

The pids limit counted threads, not just processes, and the main thread used the hundredth slot. Without the cap, the same script started 5,000 threads and stopped cleanly.

Notice that the program did not crash. It caught the error and reported a count. The Kinesis servers were not so lucky: the work that needed new threads was building the shard map, so routing stopped.

AWS's fixes fall into three groups: cut thread use now, speed up cold starts, and split the front-end fleet into cells. It also changed the services that broke alongside Kinesis.

The list below follows the summary's own words and tense. Some items were done when it was written; most were plans. This guide reports the summary only, not later progress.

• Bigger servers, very short term. Move to larger CPU and memory servers, so the fleet needs fewer servers and each keeps fewer threads. AWS said this gives significant headroom in thread count.
• Thread alarms. Add fine-grained alarming for thread consumption in the service.
• A tested higher limit. Finish testing an increase in the thread count limits in the operating system configuration.
• Faster cold starts. Move the front-end server cache to a dedicated fleet, as part of changes to radically improve cold-start time.
• A separate front-end for big users. Move a few large AWS services, like CloudWatch, to a separate, partitioned front-end fleet.
• Cells, medium term. Greatly accelerate the cellularization of the front-end fleet, meaning splitting it into cells, to match what was done with the back-end.
• Cognito, done. Its webservers were modified to sustain Kinesis API errors without exhausting their buffers.
• CloudWatch, done in US-EAST-1. Keep 3 hours of metric data in a local store, with global deployment planned in the following weeks.
• Status page practice. Support engineers are now regularly trained on the backup tool for posting updates.

A cell is an isolated copy of a service that handles only part of the traffic and shares no state with other cells. AWS guidance puts a thin cell router in front, which picks a cell using a partition key such as a customer id.

Cells attack the Kinesis problem at its root. A server in a front-end cell would only need to know the servers in its own cell. If a cell's size is capped, its thread count stays inside a range you can test. The AWS report gives the same reason: cells keep the shard-map cache within a previously tested and operated range.

AWS's cell guidance makes the same point: cap the maximum size of a cell, and keep cells small enough to test at full scale. Its simple example: if 10 cells serve 100 requests and one fails, 90 percent are unaffected.

The diagram shows a general cell layout from that guidance, not the real Kinesis design, which the summary does not describe. Notice that no cell talks to another cell.

You cannot fix a provider's outage, but you can stop it from spreading through your own code. The main moves are a capped buffer, side features that fail quietly, and checks that live outside the failing region.

• Find any per-server cost that grows with fleet size: threads, connections or memory per peer. Adding servers can weaken every server.
• Watch hidden limits such as threads, open files and memory, and alarm well before them.
• Know your cold-start time and how many servers can safely restart together.
• Make best-effort paths truly best-effort: a metrics or logging path must drop data rather than block the main job.
• Keep your status tools free of the systems they report on.

A buffer that never blocks

In the scenario, the team's API servers waited on every Kinesis write, much like Cognito's webservers. The fix is a bounded buffer: a fixed-size store that drops the oldest events when full and never makes a request wait.

Here is a small, runnable version. send_batch stands in for the code that writes a batch to Kinesis and returns True if it worked. The bottom half tries it with a stream that is down, then back.

from collections import deque

class EventBuffer:
    def __init__(self, max_events):
        self.events = deque(maxlen=max_events)  # oldest dropped when full
        self.dropped = 0

    def add(self, event):  # never waits on the stream
        if len(self.events) == self.events.maxlen:
            self.dropped += 1
        self.events.append(event)

    def flush(self, send_batch, size=100):
        while self.events:
            batch = [self.events.popleft() for _ in range(min(size, len(self.events)))]
            if not send_batch(batch):  # stream still failing
                self.events.extendleft(reversed(batch))  # back at the front, same order
                return False
        return True

# Try it: a tiny buffer, a stream that is down, then a stream that is back
buf = EventBuffer(max_events=5)
for i in range(8):
    buf.add(f"click-{i}")
print(list(buf.events), "dropped:", buf.dropped)

print("flush while down:", buf.flush(lambda batch: False, size=2))
print(list(buf.events))

sent = []
print("flush when back:", buf.flush(lambda batch: sent.extend(batch) or True, size=2))
print(sent, "left:", len(buf.events))

What just happened? On Python 3.13.16 it printed the last five clicks, click-3 to click-7, with dropped: 3. The flush while down printed False and left the same five in order. The flush when back printed True and sent all five.

A deque is a Python queue that is fast at both ends. Its maxlen caps how many events it holds, so memory stays bounded and a long outage cannot starve the server, as happened on Lambda's hosts. The dropped counter turns silent loss into a number you can chart.

One trap remains if add and flush run on different threads. If writers refill the buffer while a failed batch is out, extendleft pushes the newest events off the far end. The dropped counter never sees them.

Our probe showed it: two new events vanished and the counter stayed at 0. The fix we tested: one lock shared by add, the pops and the put-back, never held during send_batch. So add may wait briefly for the lock, but never for the network.

On put-back, return only as many events as still fit, dropping the oldest, and add the rest to dropped. In the same probe, the counter then read 2 and the newest events survived.

Tell it as one short chain of causes, then turn each link into a design choice. Interviewers care most about why a small change spread so far.

1. 1

   Name the trigger

   A small capacity addition to the Kinesis front-end fleet in US-EAST-1, on 25 November 2020. Say it was the trigger, not the root cause.

2. 2

   Explain the mechanism

   Each front-end server kept operating system threads for every peer. More servers meant more threads on every server, until all of them passed the limit.

3. 3

   Show the effect

   Servers could not finish building their shard maps, so they could not route requests. Puts and gets failed.

4. 4

   Explain the slow recovery

   Rebuilding the cache competed with live traffic, so servers came back at only a few hundred per hour.

5. 5

   Trace the cascade

   Cognito, CloudWatch, Lambda, Auto Scaling, EventBridge, ECS and EKS depended on Kinesis directly or indirectly. Even the status page tool depended on Cognito.

6. 6

   Give the fixes

   Bigger servers, thread alarms, a dedicated cache fleet, and capped cells that bound how many peers each server must know.