Retry, Backoff and Jitter: How to Avoid a Retry Storm

When every client retries a failed call at once, the retries become the outage. A fault that lasted seconds keeps the service down for many minutes, because the retry traffic never gives it room to recover.

This failure has a name: a retry storm. A retry is simply sending the same request again after it fails. A storm is thousands of clients doing it in the same second.

Picture a common setup. A team runs a weather data API that farm apps and delivery apps call for forecasts. At 6 am on a monsoon morning, traffic peaks at about 4,000 requests per second. The setup is made up, but the failure pattern is real.

At 6:02 the forecast database fails over, meaning it switches to a standby copy kept in sync. The switch takes 30 seconds, and every forecast call fails during that time.

Thirty seconds of errors should have been a small blip. Instead the API stayed down until 6:41. The graphs showed about 24,000 requests per second, six times normal, while the database was already healthy again.

The cause was retries at three places. The team's client library, the code partner apps use to call the API, retried every failed call three times with no wait. Many partner backends wrapped that library in three retries of their own.

The API gateway, the front door that passes requests on to the services, also retried failed calls three times. So one farmer's tap during the blip became a burst of calls, all landing in the same second.

The API servers ran out of connections and failed new requests too. Those failures caused more retries, which caused more failures. Follow that loop in the diagram: the database starts it, but is not part of it.

The on-call engineer ended the storm by hand. Retries alone would never have stopped, so the servers needed room first.

1. 1

   Shed load at the edge

   At 6:35 the load balancer, the box that spreads traffic across servers, was set to reject most traffic with a quick error. Cheap rejections let the API servers breathe.

2. 2

   Let the servers recover

   Open connections drained away. The forecast cache, which keeps recent answers in memory, filled up again.

3. 3

   Open the gates slowly

   Traffic came back in steps of about 20 percent, with error rates checked between steps. By 6:41 everything was normal.

Google's SRE book (site reliability engineering) gives the same advice for this state. You must sharply reduce or remove the load until the retries stop and the backends settle.

Retry backoff jitter is a three-part rule for trying a failed call again. Retry only errors that can pass, wait longer after each failure, and add randomness to every wait.

• Retry: send the same request again, but only for errors that may clear up and only when repeating is safe.
• Backoff: wait before each retry, and let the longest allowed wait grow after every failure.
• Jitter: pick each wait at random below that limit, so clients that failed together stop retrying together.

Many people picture an instant retry as harmless, even helpful. That model is wrong under load. An instant retry is one more call to a service that is already failing, sent at the worst moment.

Each part fixes a different failure from the storm above. Choosing what to retry stops pointless calls. Backoff gives the service time. Jitter stops thousands of clients from arriving in the same millisecond.

A related older name is the thundering herd. Many waiting processes or clients wake at the same moment and rush at one resource, and most of them fail. Jitter is a standard cure for both.

Retry only when the error is likely to be temporary and repeating the request is safe. Everything else should fail fast, with a clear message, on the first try.

A transient error is one that may pass on its own, like a timeout, a dropped connection or a briefly overloaded server. A permanent error fails the same way every time, like a request with a missing field.

• Retry: timeouts and dropped connections. The network or the server may be fine a moment later.
• Retry: 503 Service Unavailable, which HTTP defines as a temporary overload or maintenance that should pass after some delay.
• Retry: 502 Bad Gateway and 504 Gateway Timeout. A gateway, a server that passes requests on, got a bad answer or no answer from the service behind it.
• Retry, but slowly: 429 Too Many Requests, which means a rate limiter (a guard that caps how often you may call) is turning you away.
• Do not retry: 400 Bad Request, 401 Unauthorized, 403 Forbidden and 404 Not Found. The same request will almost always get the same answer.
• Take care with 500 Internal Server Error. It often comes from a bug, and the work may have half run, so retry it once at most.

Both of the main sources draw this line. Google's SRE book says not to retry permanent errors or malformed requests, because neither will ever succeed. Marc Brooker's article on timeouts and retries says the same about client errors.

Safe to repeat has a precise name: idempotent. A call is idempotent when doing it twice has the same effect as doing it once. HTTP defines PUT, DELETE and the read-only methods, such as GET, as idempotent. POST is not on that list.

The HTTP standard (RFC 9110) says a client should not automatically retry a non-idempotent request. The exception is when it knows a repeat is harmless, or that the first request never took effect. Payments solve this with idempotency keys, which have a guide of their own.

The partner apps in our setup only read forecasts, so their calls are safe to repeat. The real question for them is how often and how fast.

Exponential backoff means letting the wait before each retry grow, usually doubling, up to a fixed maximum called the cap. It gives a struggling service more and more room to recover.

The team's new client library uses a base of 0.2 seconds. The first retry waits up to 0.2 seconds, then up to 0.4, 0.8 and 1.6. Without a cap, the ninth retry could wait up to 51.2 seconds.

As a formula, the limit for retry number n is min(cap, base × 2 to the power n), with n starting at 0. The min means take whichever is smaller. With a 5-second cap, no wait goes above 5 seconds.

Backoff needs two more limits. A maximum number of attempts stops one request from retrying forever. A total deadline stops the whole call from taking longer than the user will wait.

Google's SRE book gives a concrete number. Its chapter on handling overload allows up to three attempts per request. After the third failure, the error goes back to the caller instead of being retried again.

Backoff alone is still not enough, and a New Year's Eve phone call shows why.

Exponential backoff needs jitter because clients that failed together will retry together if they all wait the same time. Jitter makes each wait random, so retries spread out instead of arriving in waves.

Picture midnight on New Year's Eve. Everyone in Bhopal calls their family at 12:00, and the mobile network says all lines are busy. Everyone presses redial at once, so the lines stay jammed.

Some people get clever and decide to wait five minutes. But thousands of them chose five minutes. At 12:05 the same crowd dials together, and the lines jam again. That is backoff without jitter.

Your aunt does something different. She waits a random few minutes, maybe three, maybe seven. Her call goes through on the first redial, because she stepped out of the crowd. That is jitter.

Where the picture breaks: a phone network never tells callers how long to wait. A server can, with the Retry-After header covered later. Software clients also fail within milliseconds of each other, so their crowd is far tighter.

The partner apps are the crowd. Thousands of them failed in the same 30 seconds. With plain backoff, every one waits exactly 0.2 seconds, then 0.4, then 0.8, and each wave hits the API together.

Marc Brooker measured this in 2015 with a simulation of clients competing to update one database row. Backoff alone helped only a little, because calls still bunched up. Adding jitter cut the work sharply.

Our own re-run of his model, shown later, agrees: with 100 clients, full jitter made fewer than half the calls of plain backoff. Google's SRE book is just as firm: always use randomised exponential backoff when scheduling retries.

Write one loop that checks the error, works out the backoff limit, picks a random wait below it, and checks the attempt limit, deadline and budget before sleeping. Every check has a clean way to give up.

Here is that loop in Python. send is any function that makes one API call, with its own timeout so one slow try cannot overrun the deadline. ApiError stands in for the error your HTTP library raises, holding the status, a timed_out flag and any Retry-After value.

import random
import time

RETRYABLE = {429, 502, 503, 504}

class ApiError(Exception):
    def __init__(self, status=None, timed_out=False, retry_after=None):
        super().__init__(status)
        self.status = status            # HTTP status, or None
        self.timed_out = timed_out      # no reply arrived in time
        self.retry_after = retry_after  # seconds from Retry-After, or None

def fetch_with_retries(send, budget=None, max_attempts=4,
                       base=0.2, cap=5.0, deadline=3.0):
    start = time.monotonic()
    if budget is not None:
        budget.record_request()
    for attempt in range(max_attempts):
        try:
            return send()
        except ApiError as err:
            if not (err.timed_out or err.status in RETRYABLE):
                raise                   # this error cannot pass
            if attempt == max_attempts - 1:
                raise                   # out of attempts
            ceiling = min(cap, base * 2 ** attempt)
            wait = random.uniform(0, ceiling)   # full jitter
            if err.retry_after:
                wait += err.retry_after         # server's pause, plus jitter
            if time.monotonic() - start + wait > deadline:
                raise                   # would miss the deadline
            if budget is not None and not budget.can_retry():
                raise                   # retry budget spent
            time.sleep(wait)

What just happened? The first two raise lines give up at once on errors that cannot pass, and on the last attempt. The loop leaves 500 out of RETRYABLE, the safest choice. The call to random.uniform is full jitter: any wait from zero to the limit is equally likely.

With these defaults, one call makes at most four tries and sleeps at most 1.4 seconds in total, unless the server sends Retry-After. That is 0.2 plus 0.4 plus 0.8 seconds, the three possible waits.

The Retry-After line adds the server's pause to the jittered wait. A plainer max(wait, retry_after) would make every client sleep exactly the server's value, and they would all return in the same instant.

Retry-After can also be a date. Convert it to seconds before you store it on the error. The deadline check counts the time spent so far plus the next wait, so a long pause makes the call give up early instead.

That short total suits a farm app, where a farmer is staring at the screen. A nightly data sync could use a bigger cap and more attempts, because nobody is waiting on it.

Check that jitter really breaks up the crowd

You can watch the difference on your own laptop. This toy model has 2,000 clients that failed together and a service that answers only 25 calls per 10 milliseconds. It has no network delay. Pause and predict: how many calls will 2,000 clients make with and without jitter?

import random

def herd(jitter, clients=2000, per_slot=25, base=0.2, cap=5.0):
    """All clients failed together at time 0. The service can answer
    per_slot calls in each 10 ms slot; the extra calls fail again."""
    rng = random.Random(7)
    due = {c: 0.0 for c in range(clients)}  # client -> next call (s)
    tries = dict.fromkeys(due, 0)
    calls, peak, ms = 0, 0, 0
    while due:
        now = [c for c, t in due.items() if t < (ms + 10) / 1000]
        rng.shuffle(now)
        calls += len(now)
        if ms > 0:
            peak = max(peak, len(now))      # busiest retry slot
        for c in now[per_slot:]:            # over capacity: fail
            ceiling = min(cap, base * 2 ** tries[c])
            wait = rng.uniform(0, ceiling) if jitter else ceiling
            due[c] = ms / 1000 + wait
            tries[c] += 1
        for c in now[:per_slot]:            # the rest succeed
            del due[c]
        ms += 10
    print(f"jitter={jitter}: {calls} calls, busiest 10 ms slot "
          f"{peak} calls, all done after {ms / 1000:.2f} s")

herd(jitter=False)
herd(jitter=True)

On Python 3.13.16 it prints two lines. Without jitter: 81000 calls, busiest 10 ms slot 1975 calls, all done after 376.21 s. With jitter: 6487 calls, busiest slot 205 calls, done after 3.45 s.

The no-jitter clients move in lockstep, so each wave of about 2,000 calls lets only 25 through. Real networks add small random delays that blur the waves a little, so treat these numbers as the toy model's, not a forecast.

Pick full jitter unless you have a reason not to. It is the simplest, and in Brooker's 2015 simulation it made the fewest calls. Equal and decorrelated jitter are the two common alternatives.

We re-ran that model with our own code on Python 3.13.16, averaging 100 runs of simulated time. At 100 clients, plain backoff averaged about 1,850 calls and full jitter about 795. Decorrelated jitter made about 1,000 calls but finished a little sooner than full jitter.

Full jitter needs no memory between tries, so it is the easiest to get right. Decorrelated jitter has to remember the last wait, which is one small extra piece of state per call.

Equal jitter makes sense only when a near-zero wait is useless, because it never waits less than half the limit. Most clients do not have that need.

Some systems use a narrower band. gRPC, a popular open-source framework for calls between services, multiplies each retry backoff by a random number between 0.8 and 1.2.

That is gRPC's retry design, gRFC A6, last updated on 23 August 2024. The grpcio 1.84.0 source, the latest stable release as of 19 October 2026, uses the same 0.2 jitter. Its core library also treats any maxAttempts above 5 as 5.

A narrow band keeps waits predictable, but it spreads clients less. For a crowd that failed together, as the partner apps did, a wider random range does more good.

Retries multiply because every layer that retries repeats the whole call below it, including that layer's own retries. If three layers each make up to four tries, the bottom service can see 4 × 4 × 4 = 64 calls.

Google's SRE book uses this exact example. In its chapter on cascading failures, three layers each retry three times, so one user action can create 64 attempts on the database.

Brooker's article describes a five-deep stack of service calls with three retries at each layer. He counts three tries per layer, so the database load grows 243 times, 3 to the power 5. Count four tries per layer, as above, and it is 1,024.

The weather storm had the same shape. The diagram shows one farmer's tap during the failover. Read the note on each box and watch the number grow as you go down.

The fix is to pick one layer that retries. Google's SRE book says requests should only be retried at the layer directly above the one rejecting them. Brooker also advises retrying at a single point in the stack.

The team chose the client library as its only retry point. It sits closest to the partner and knows how long the user can wait. The gateway stopped retrying and now returns errors straight away.

Partners were asked to remove their own retry wrappers, and the library's guide now says so in its first paragraph. One tap now means at most four calls, not 64.

A retry budget limits how many retries a client may send compared with its normal requests, such as 10 percent. When the budget is spent, failures go back to the caller at once instead of being retried.

Brooker calls retries selfish. When a client retries, it spends more of the server's time to raise its own chance of success. That is fine for a rare blip, but harmful when the server is already overloaded.

Backoff and jitter spread retries over time, but they do not cut how many there are. During a long outage, every request still makes its full set of tries. A budget caps the total.

Google's SRE book describes a per-client retry budget: a request is retried only while retries stay below 10 percent of that client's requests. It pairs this with the three-attempt limit per request.

Its worst case is a datacenter that rejects most requests. With only the three-attempt limit, traffic grows to just below 3 times. With the 10 percent budget added, growth stays near 1.1 times in the general case.

One simple way to build a budget is a counter of credits. Every new request earns 1 credit, up to 100. Every retry spends 10. Whole numbers avoid rounding errors: adding 0.1 to a running total ten times in Python gives 0.9999999999999999, not 1.

class RetryBudget:
    """About one retry per 10 requests, plus a burst of 10 to start."""

    def __init__(self, cost=10, max_credits=100):
        self.cost = cost                # credits one retry spends
        self.max_credits = max_credits
        self.credits = max_credits      # start full

    def record_request(self):
        self.credits = min(self.max_credits, self.credits + 1)

    def can_retry(self):
        if self.credits >= self.cost:
            self.credits -= self.cost
            return True
        return False

The counter starts full, so a short blip can still use 10 retries straight away. In a long outage it empties. After that, a client gets about one retry for every 10 new requests.

The cap of 100 matters. It stops a quiet hour from saving up a huge pile of credits, which would allow a burst of retries later. The loop above calls record_request once per request and can_retry before each retry.

To check it, put the retry loop, the RetryBudget class and these lines in one file, and run it with Python 3. They send 1,000 requests to a service that always answers 503, first without a budget and then with one.

time.sleep = lambda seconds: None    # skip real waiting in this test

calls = 0
def always_503():
    global calls
    calls += 1
    raise ApiError(status=503)

for budget in (None, RetryBudget()):
    calls = 0
    for _ in range(1000):
        try:
            fetch_with_retries(always_503, budget)
        except ApiError:
            pass
    print("budget" if budget else "no budget", calls, "calls")

On Python 3.13.16 it prints no budget 4000 calls, then budget 1109 calls. That is 1,000 first tries plus 109 retries: 10 from the full counter and about one per 10 requests after that.

gRPC builds the same idea in, called retry throttling. Each call that fails with a status the policy marks retryable removes a token, and each success adds back a configured fraction, such as 0.1 in the design's example. When the count falls to half the maximum or below, retries stop.

A server tells clients to back off by returning a clear status, such as 503 or 429, with a Retry-After header that says how long to wait. A header is a labelled line of extra information sent with the reply.

Retry-After is defined in RFC 9110, section 10.2.3. Its value is a date or a whole number of seconds, so Retry-After: 120 means wait two minutes. RFC 6585 lets a 429 reply carry it too.

In our setup, the API now sends 503 with Retry-After: 2 while the database fails over. Retry-After counts whole seconds, and the API sends every client the same 2. That is why the loop adds jitter on top of it.

Google's SRE book adds one more signal. A backend that decides a request should not be retried returns an overloaded, do not retry error. The layer above then passes the failure up instead of repeating it.

gRPC has a version of this called pushback. The server sets grpc-retry-pushback-ms, a wait in milliseconds. A negative value tells the client not to retry at all.

gRPC waits exactly the pushback delay, with no jitter added. Both gRFC A6 and the grpcio 1.84.0 source do this, as of 19 October 2026. If you send pushback to many clients at once, spread the values yourself.

Jitter your timers too

Retries are not the only thing that arrives in waves. Many farm apps refreshed forecasts on a timer at exactly 6:00 every morning, so the API saw a spike at 6:00:00 sharp, every day.

Brooker's article recommends adding jitter to all timers and periodic jobs, not just retries. For scheduled work, it uses a fixed offset per host instead of a fresh random one, so any overload repeats in a pattern you can debug.

The team asked partners to refresh at a random minute between 5:50 and 6:10, fixed for each device. The sharp 6 am spike flattened into a gentle hump.

Explain it as limits on a selfish habit, and show which failure each limit fixes. Start with the storm, then add the fixes in this order.

1. 1

   Name the failure

   Start with a short outage and thousands of clients. Show how instant retries turn a 30-second blip into a long outage.

2. 2

   Choose what to retry

   Retry timeouts, 502, 503, 504 and 429. Fail fast on other client errors, and retry only calls that are idempotent.

3. 3

   Add capped exponential backoff

   Give the base, the doubling and the cap. Add a limit of three or four attempts and a total deadline.

4. 4

   Add jitter

   Say full jitter, a random wait from zero to the backoff limit. Explain that it breaks the sync between clients that failed together.

5. 5

   Retry at one layer

   Count the layers and multiply. Pick one retry point, usually the one closest to the caller, and turn the others off.

6. 6

   Cap the total

   Add a retry budget of about 10 percent and honour Retry-After with jitter on top. Mention a circuit breaker for outages that last minutes, not seconds.