Backpressure Explained: Slowing Producers Before Queues Explode

The queue grows by the gap between work arriving and work leaving, every second, until the server runs out of memory or the work goes stale. Nothing tells the senders to slow down.

Picture a common setup. A bulk SMS gateway in Jaipur sends one-time passwords (OTPs) and offer messages for banks and delivery apps. The setup is made up, but the problem is real.

The gateway's API servers, the machines that take requests from other programs, can accept about 6,000 messages per second. Its contract with the mobile carrier, the phone network that delivers the texts, allows only 1,500 messages per second (made up).

On most days that gap never matters. Traffic is a few hundred messages per second, and the carrier link is mostly idle.

Then a bank starts a festival campaign at 9 in the morning. It sends 2 million (20 lakh) offer messages as fast as it can call the API. The gateway says yes to every one, puts it in a list in memory and replies with success.

• By 9:05, about 1.35 million messages are waiting in that list.
• OTPs from a delivery app land at the back of the same list, behind the offers.
• Each OTP expires in 5 minutes, but one that arrives at 9:05 would wait about 15 minutes before it is even sent.
• At 9:06 the server runs out of memory and restarts. Every message still in the list is lost.

Customers could not log in to their bank or receive their parcels. The carrier link was never the bug. The bug was that the gateway kept saying yes long after it could keep its promise.

This guide adds backpressure to the gateway one step at a time, and says what each choice costs.

Backpressure is a way for a slow consumer to tell a faster producer to slow down, so work does not arrive faster than it can be finished. The signal flows backwards, against the direction of the work.

• Producer: the part that creates work, like the bank's campaign servers.
• Consumer: the part that does the work, like the gateway's sender process that talks to the carrier.
• Buffer: the waiting area between them. It is usually a queue, where items leave in the order they arrived.

Picture a small goli soda unit, where bottles are sealed with a glass marble. One machine fills bottles. The next station presses in the marble and seals each one. The filler is fast and the sealing station is slow.

Filled bottles wait on a steel table before sealing. If the filler keeps going at full speed, the table overflows and the fizz goes flat. So the filler operator watches the table, and pauses when it is full.

Map it part by part. The filler is the producer, the sealing station is the consumer, and the table is a buffer with a fixed size. The full table is the backpressure signal, and the operator's pause is the producer slowing down.

Where the picture breaks: the operator can see the table. In software, the producer usually sits on another machine and sees nothing. Unless you build a signal, such as a full queue or an error reply, it never learns.

Here is the gateway on the day it crashed. Notice that every arrow points forward. No part of the system can tell the bank to wait.

A queue grows by the arrival rate minus the processing rate, every second, for as long as the gap lasts. It can soak up a short burst, but it cannot fix a lasting mismatch.

Do the sum for the gateway. Messages arrive at 6,000 per second and leave at 1,500 per second, so the queue grows by 4,500 every second. After 5 minutes, that is 4,500 times 300, or 1.35 million waiting messages.

Pause and predict: a new OTP joins the back of that queue at 9:05. How long does it wait before the sender reaches it?

The sender clears 1,500 messages per second, so 1.35 million messages take 900 seconds to drain. That is 15 minutes, three times longer than the OTP stays valid.

That sum is plain arithmetic for a first-in, first-out queue: the newest item waits about queue length divided by processing rate. A queue in a steady state obeys a related rule called Little's law.

Little's law says the average number of items in a system equals the average arrival rate times the average time each item spends there. At the same arrival rate, halve the time each item spends waiting, and you halve the queue.

So a long queue is not storage. It is delay. Every extra message in the queue adds waiting time to every message behind it.

There are only two ways out of a lasting mismatch. You make the consumer faster, or you let less work in by slowing the producer, refusing work or dropping it. Backpressure is the second one, done on purpose with a clear signal.

There are five common ways: buffer within a limit, block the producer, reject or drop extra work, let the consumer pull work, and add more consumers. Most systems combine two or three of them.

• Bounded buffer. Keep a queue, but give it a size limit, so it absorbs bursts and then stops growing.
• Block the producer. When the queue is full, the producer waits until there is room.
• Reject or drop. Refuse new work with a clear error, or throw away the least useful work. Dropping work on purpose is called load shedding.
• Pull instead of push. The consumer asks for work only when it is ready, so it takes on no more than it asked for.
• Scale out. Add consumers until they keep up. This only works if the slow part can actually grow.

The last point mattered for the gateway. The gateway's team could add ten more sender servers, but the carrier would still accept 1,500 messages per second. When the slow part belongs to someone else, more workers only move the queue.

A bounded queue creates backpressure by making the producer wait whenever the queue is full. In Python's asyncio library, which runs many tasks on one thread, a single line of execution, you get this by giving asyncio.Queue a maxsize.

The Python documentation says a queue whose maxsize is zero or less is infinite, and zero is the default. That is the trap the gateway fell into. With a positive maxsize, await put() blocks once the queue is full, until get() removes an item.

Here is a small, runnable version of the sender process. SlowCarrier stands in for the carrier link and takes about 1 millisecond per message. The queue holds at most 100 waiting messages.

import asyncio
import time

class SlowCarrier:
    """Stand-in for the carrier link: about 1 ms per message."""
    async def send(self, msg):
        await asyncio.sleep(0.001)

async def producer(queue, messages, stats):
    for msg in messages:
        await queue.put(msg)            # waits here while the queue is full
        stats["peak"] = max(stats["peak"], queue.qsize())

async def sender(queue, carrier_link):
    while True:
        msg = await queue.get()
        await carrier_link.send(msg)    # the slow step
        queue.task_done()

async def main():
    queue = asyncio.Queue(maxsize=100)  # at most 100 waiting
    stats = {"peak": 0}
    start = time.perf_counter()
    worker = asyncio.create_task(sender(queue, SlowCarrier()))
    await producer(queue, range(2000), stats)
    await queue.join()                  # wait until all are sent
    worker.cancel()
    print("peak queue size:", stats["peak"])
    print(f"sent 2000 messages in {time.perf_counter() - start:.1f} s")

asyncio.run(main())

Run it with python3. The first line should read peak queue size: 100, which does not depend on timing. The second line's time depends on your system's timer. Our runs on Python 3.13.16 on Linux took 2.5 to 2.6 seconds.

Now change maxsize=100 to maxsize=0 and run it again. The peak queue size becomes 2000, because the producer never waits. That is the gateway's crash in miniature.

What just happened? The await queue.put(msg) line is where backpressure lives. When the queue is full, the producer pauses on that line, so it slows to the sender's pace with no timer or rate setting.

Blocking works well inside one process. It is a poor fit across a network. If the gateway's API held every bank request open while the queue was full, thousands of open connections would pile up on the API servers instead.

You refuse extra work quickly, with an HTTP status code and a Retry-After header that says how long to wait, in seconds or as a date. The caller then waits and tries again, instead of you holding the work.

• 429 Too Many Requests. RFC 6585, an internet standards document, defines it as the user having sent too many requests in a given amount of time. Use it when one client is over its own limit.
• 503 Service Unavailable. It means the server cannot handle the request right now, for example because it is overloaded. Use it when a shared resource is full, whoever is calling.
• 202 Accepted. It means the request was accepted for processing, not that the work is done. HTTP has no way to send a later response with the outcome, so the caller must check back some other way.

The team first splits the traffic in two, because OTPs and offers need very different treatment. An OTP is useless after a few minutes, so it must go fast or not at all. An offer can wait an hour and still be fine.

This diagram shows the decision the API makes for every new message. Follow one OTP and one offer through it, and notice that each refusal has its own status code.

Here is the OTP lane in code. Unlike put(), the put_nowait() method never waits. The Python documentation says it raises QueueFull when no free slot is available, and the gateway turns that exception into a 503.

import asyncio

CARRIER_RATE = 1500        # messages per second we may send (made up)
MAX_WAIT_SECONDS = 10      # an OTP that waits longer is no use
otp_lane = asyncio.Queue(maxsize=CARRIER_RATE * MAX_WAIT_SECONDS)

def accept_otp(msg):
    try:
        otp_lane.put_nowait(msg)       # never waits
    except asyncio.QueueFull:
        return 503, {"Retry-After": "5"}
    return 202, {}

The maxsize line turns a business rule into a limit: 1,500 per second times 10 seconds is 15,000 OTPs. An OTP that gets in waits at most about 10 seconds, as long as the sender keeps up its 1,500 per second.

To check it, call accept_otp 15,001 times without a sender running. The first 15,000 calls return 202. The 15,001st returns 503 with Retry-After set to 5, and one get_nowait() frees a slot again.

A fast, honest no beats a slow, silent failure. During the crash, the delivery app got success replies for OTPs that would never arrive in time. With a 503, the app knows to retry later or offer a voice call instead.

The team also changes the client contracts. Each bank gets a per-second rate limit, and big campaigns go in as scheduled jobs. The rate limit stops one client from filling everything, and backpressure handles whatever still gets through.

All three let the receiver control how much it gets. Kafka consumers pull data when ready, RabbitMQ caps unacknowledged messages per consumer, and TCP tells the sender how many bytes the receiver will accept.

Kafka: the consumer pulls

Kafka is an event streaming system. It stores messages in topics, named streams split into partitions. Each partition is an ordered log kept on servers called brokers. The Kafka 4.3 design documentation says producers push data to the broker and consumers pull it.

The same page explains why. In a push system the broker controls the rate, so a consumer that reads slower than data is produced tends to be overwhelmed. With pull, a slow consumer falls behind and catches up when it can.

So the team moves campaign offers into a Kafka topic. The 2 million offers wait in the brokers' log, not in the API servers' memory. The gap now shows up as consumer lag: messages written but not yet read.

Each poll() call returns at most max.poll.records records, 500 by default, as of Kafka 4.3. A fetch is also limited by fetch.max.bytes, 50 mebibytes (MiB) by default. One record batch bigger than that is still returned, so the consumer can make progress.

This diagram shows the pull loop between one sender and the campaign store. Notice that the sender starts every round. The store sends nothing until it is asked.

Lag is not free storage. As of Kafka 4.3, log.retention.hours defaults to 168, so unless you override it, brokers delete log data after about 7 days. A consumer that falls further behind than that never sees the deleted messages.

RabbitMQ: a cap on unacknowledged messages

RabbitMQ is a message broker that usually pushes messages to consumers, so it needs a different brake. That brake is the prefetch count: how many messages a consumer may hold without acknowledging, meaning confirming it has finished them.

The RabbitMQ 4.4 documentation says a prefetch value of 0 is treated as infinite. With the global flag set to false, RabbitMQ applies the limit to each consumer. In plain AMQP 0-9-1, the protocol RabbitMQ speaks, that setting shares it across a channel.

Two details catch people out, as of the 4.4 docs. With automatic acknowledgements there is no limit at all, and consumers can be overwhelmed. Prefetch also has no effect on messages fetched with basic.get, the polling API.

TCP: the receive window

TCP is the transport under HTTP/1.1 and HTTP/2. Every TCP segment carries a window field. RFC 9293, the current TCP standard, says it is the number of bytes the sender of that segment is willing to accept.

When the receiving program stops reading, its buffer fills and the window it advertises falls to zero. The sender stops sending new data. RFC 9293 says it must keep probing, with at least one byte of new data if it has any.

The standard also says the first probe should wait one retransmission timeout, the time TCP waits before resending lost data, and later probes should back off exponentially. So backpressure reaches your code even if you never planned for it. Once the sender's own buffer also fills, a blocking send() call simply waits.

We checked this over loopback, a connection from one Linux machine (kernel 6.18) to itself. A receiver that never read made a non-blocking sender stop after about 3.9 megabytes (MB) of buffered data. Once the receiver read, the sender could send again. Buffer sizes vary by system.

Reactive Streams: backpressure as a standard

Reactive Streams is a standard for passing streams of data between parts of a program without blocking. Its website says backpressure is built in so that the queues between threads can be bounded.

Java 9 added matching interfaces as java.util.concurrent.Flow. There, a subscriber calls request(n) to ask for n more items. The Java documentation says items are not issued unless requested.

It looks like a set of small, bounded lanes, each with its own brake, so no single flood can fill everything. The new design keeps OTPs fast and lets offers wait safely.

Read this diagram from left to right. Each box notes how it pushes back when busy. The monitoring box watches the two places where work can wait.

Now replay the festival campaign. If the bank's contract allows 6,000 per second, its 2 million offers reach Kafka in about 6 minutes. The sender keeps OTPs moving first and fills the spare carrier capacity with offers.

Say OTPs use about 300 of the 1,500 slots per second (made up). Offers then get about 1,200 per second, so 2 million take roughly 28 minutes. Every offer is sent, and no OTP waits more than about 10 seconds.

The team can now tell the bank something honest before the campaign starts: 2 million offers take about half an hour. If the bank needs them faster, that is a talk about buying more carrier capacity, not a crash at 9:06.

Backpressure goes wrong when the signal stops halfway, when it blocks the wrong thing, or when the caller ignores it. Each mistake moves the queue somewhere you are not watching.

• Blocking across the network. Holding requests open while full just moves the pile into open connections and threads. Refuse fast at the edge instead.
• Buffers at every layer. Ten seconds of buffer in each of six services adds up to a minute of hidden delay. Know every place where work can wait.
• One shared lane for everything. When offers and OTPs share one queue, a flood of bulk work blocks the urgent work. Give urgent work its own lane.
• Limits set in items, not time. A limit of one million sounds safe until you see it is about 11 minutes of waiting at 1,500 per second.
• Callers that retry instantly. A refusal with no Retry-After and no jitter invites a retry storm. Tell callers when to come back.
• No alerts on lag or refusals. A rising 429 or 503 rate, or growing consumer lag, is the early warning. If nobody watches, angry clients tell you first.

Explain it as a rate mismatch first, then show where the slow part's signal travels and what each caller does when it hears it.

1. 1

   Find the slowest part

   Name the bottleneck and its rate, like a carrier link at 1,500 per second. Say whether you own it or can scale it.

2. 2

   Do the queue sum

   Show the growth: arrival rate minus processing rate, times duration. Turn queue length into waiting time by dividing by the processing rate.

3. 3

   Bound every buffer

   Give each queue a limit based on the longest acceptable wait. Say what happens when it is full.

4. 4

   Pick a brake for each hop

   Use pull or a prefetch limit between internal services, blocking inside a process, and fast 429 or 503 replies at the public edge.

5. 5

   Separate urgent from bulk

   Give time-critical work its own small lane, and send bulk work to storage on disk where it can wait, within its retention limit.

6. 6

   Watch it

   Alert on queue depth, consumer lag and the refusal rate. These show trouble long before memory runs out.