Inngest's October 2026 Outages Explained: When Deadlines Drained the Pool

Giving up by closing the connection feels free, but it is not. If the query is still waiting in a connection pooler, the pooler may not notice you left. It can later spend a real database connection on your abandoned request and then have to close it.

This is the core of the Inngest outage of October 2026. Inngest runs background jobs for other companies, and its official report explains how short deadlines on database lookups left two PgBouncer poolers with no database connections.

Picture a common setup. A job-scheduling service checks job state in PostgreSQL through PgBouncer, and every lookup has a 50 millisecond deadline so a slow database cannot stall a worker. At the top of every hour, scheduled jobs wake up together and the query rate roughly triples. The setup is made up, but the failure is the one Inngest reported.

We will follow that service through this guide. First the words you need, then what happened, then the mechanism, which we reproduced on one machine.

You need five terms. Each one is a small idea, and the outage only makes sense when you see how they fit together.

A driver is the library your code uses to talk to the database, such as pgx for Go or psycopg for Python. Drivers differ in what they do when a deadline passes, and that difference matters here.

The Inngest outage was really three incidents in one month, each with its own published report. Two were on 9 and 10 October, and one more came on 20 October.

• 9 October, 21:05 to 21:49 UTC: one data-centre provider lost network links to Inngest's other data centres. Inngest says function execution depends on all three of its data centres, so execution halted.
• 10 October, 14:00 to 14:48 and 15:30 to 16:25 UTC: two PgBouncer poolers lost all their database connections and could not rebuild them. This is the main story of this guide.
• 10 October, 21:07, to 11 October, 04:17 UTC: intermittent delays, which the report links to the emergency move to other poolers.
• 20 October, 13:00 to 13:15 and 15:20 to 15:35 UTC: a loss of database connectivity, plus event-ingestion services that could not restart without the database.

The 10 October timeline in the report's own times

Notice the times: 14:00 and 15:30 are the top and the bottom of an hour. Inngest says cron work, meaning jobs scheduled to run at fixed clock times, roughly triples the query rate at those moments. A new set of lookups had also rolled out before the weekend. Hold on to both facts: the burst made queues longer, and lookups in this path had short deadlines.

PgBouncer keeps a small set of open database connections per pool and lends them out. When all of them are busy, a new query waits in a first-come queue until one comes back.

Think of a small restaurant with four tables and a waiting list. The tables are database connections, the diners are queries and the host at the door is PgBouncer. A missing table must be built, and there is only one carpenter, who builds one table at a time.

Where the picture breaks: a real host notices when a diner leaves the queue. PgBouncer may not notice until it has given that diner a table, and then it must break the table.

Two details from PgBouncer's own code shape the outage. First, the pool opens new database connections one at a time. Second, it decides what to do with a waiting client only when that client's turn comes.

The first detail is a code comment in PgBouncer 1.26.1. It sits in launch_new_connection, the function that opens database connections. It says "Allow only a single connection attempt at a time". While one connection is being opened, a request for another is skipped.

A cancel request alone does nothing, and closing the connection later costs a database connection. We checked each step on PgBouncer 1.26.1 with PostgreSQL 16 and the Python driver psycopg 3.3.6, with a pool of one connection.

Step 1: the cancel request is dropped

A cancel request names a client, and PgBouncer looks for a database connection linked to that client. A queued client has none yet, so PgBouncer drops the request with the log reason "cancel request for idle client".

In our test the query was not cancelled at all. It waited behind a 1.5 second query that had already started on the only connection. It finished 1.26 seconds after it was sent.

Step 2: the goodbye is read too late

Next we closed a client connection while its query was still queued. Closing sends a short goodbye message over the network connection, called a socket. PgBouncer had already read and parked the query, but it did not read that socket again while the client waited.

When a connection became free, PgBouncer processed the parked query, linked a database connection to it and only then read the goodbye. The database connection was now mid-request, which PgBouncer's code calls not ready. It cannot safely lend such a connection to anyone else, so it closes it.

(lines trimmed; times are seconds past 03:42)
26.294  C-...:57822 read pkt='Q' len=58     (query arrives; no free connection)
26.294  C-...:57822 pause_client             (query parked; socket not read)
        (the client closes its socket at about 26.49)
27.047  C-...:57822 activate_client          (a connection is free; its turn)
27.047  C-...:57822 linking client to S-...  (a database connection is handed over)
27.047  C-...:57822 read pkt='X' len=5      (only now PgBouncer sees the goodbye)
27.047  S-...:5434 closing because: client disconnect while server was not ready

Why drivers create this pattern

The line to watch in the trace above is the last one: a database connection closed because its client vanished. Inngest's report says that in its case "the driver cancelled the query and closed its client connection". Its services use pgx, a Go driver, where a deadline travels in a value called a context. The pgx documentation says: "The default behavior when a context is canceled is for the method to immediately return. In most circumstances, this will also close the underlying connection."

Inngest also links pgx issue 1780, about runaway connection attempts through PgBouncer. Put together, the deadline becomes a closed app connection, and the closed app connection becomes a destroyed database connection. The close costs the connection; the dropped cancel only fails to help.

It collapsed because connections were destroyed faster than PgBouncer could open new ones. Fewer connections meant longer queues, longer queues meant more missed deadlines, and more missed deadlines destroyed more connections.

Inngest describes the same loop. Its report says each pooler opens connections one at a time, connections were destroyed faster than they could be replaced, and thousands of queued clients kept missing their deadlines. The report does not name its PgBouncer version.

What our lab reproduction showed

We built our job-scheduling service in miniature on one machine. PgBouncer had a pool of 4 connections in transaction mode, and a small proxy added 50 milliseconds to every new database connection, standing in for network and login time.

Ten steady workers ran 5 millisecond queries with a 50 millisecond deadline for 15 seconds. Between second 5 and second 10, 30 more workers joined, a burst of four times the demand. The pool started with 4 open connections. All numbers below are from our lab, not from Inngest.

What just happened? The left column is the outage in miniature. Once the burst started, the pool never recovered, even when the load went back to 10 workers. In the right column, callers missed deadlines during the burst, but the database kept finishing about 700 queries per second.

Why a bigger pool and restarts did not save Inngest

Inngest says restarting the poolers did not reliably recover them, and raising the pool size did not help. Its reason: opening connections, not the pool limit, was the bottleneck.

Our lab shows why. A 20-connection pool usually rode out the burst, but still collapsed for good in 4 of 12 runs across two client versions. A higher limit adds headroom, but it does not open connections any faster once the spiral starts.

The arithmetic is simple. With one connection attempt at a time and 50 milliseconds per attempt, a pool can replace at most about 20 connections per second. A repeat of our collapsing run opened 222 connections in about 12 seconds, about 18 per second, just under that ceiling.

Our deadline equals the 50 millisecond connect time, so a query that must wait for a brand-new connection has already lost. That also makes a cold start risky: in an earlier run without pre-opened connections, the pool collapsed in the first second at the steady load of 10 workers.

Inngest's fix is to let the deadline end the caller's wait without cancelling the query. In its words, "Database lookups no longer cancel queries waiting in the pooler". The query finishes and the connection goes back to the pool.

The pattern works in any language. Run the query on a worker that owns the connection until the query is done, and let the caller wait on that worker with its own deadline. Here it is in Python with psycopg_pool 3.3.3, run against our lab setup with a 2-connection app pool.

import concurrent.futures as cf
import psycopg_pool

db = psycopg_pool.ConnectionPool("host=127.0.0.1 port=6432 dbname=app user=postgres",
                                 min_size=2, max_size=2, kwargs={"autocommit": True})
db.wait()                                  # open the pool before taking traffic
runner = cf.ThreadPoolExecutor(max_workers=2)

def run_query(sql):
    with db.connection() as conn:          # connection goes back to the pool only
        return conn.execute(sql).fetchone()  # after the query has finished

def lookup(sql, deadline_s):
    future = runner.submit(run_query, sql)
    try:
        return future.result(timeout=deadline_s)  # the caller's deadline
    except cf.TimeoutError:
        future.cancel()          # drops it only if it has not started yet
        return None              # caller gives up; a running query is not cancelled

lookup("select 1", 1.0)                    # warm-up: PgBouncer opens its server connection
print("fast:", lookup("select 'ok'", 0.05))
print("slow:", lookup("select pg_sleep(0.2), 'late'", 0.05))
runner.shutdown(wait=True)                         # let the slow query finish
stats = db.get_stats()
print("pool:", stats["pool_size"], "open,", stats["pool_available"], "available")
db.close()

Our output was "fast: ('ok',)", then "slow: None", then "pool: 2 open, 2 available". The slow caller gave up after 50 milliseconds, yet both connections were back in the pool. PgBouncer logged zero not-ready closures.

The trade-off is work you no longer need. Abandoned queries that have started still run, so they must be cheap and safe to finish. The executor's queue has no size limit, so under heavy load also cap it or fail fast. Inngest says it is reviewing its remaining database calls and their timeouts.

Driver and pooler settings to know

• pgx for Go: the documentation describes DeadlineContextWatcherHandler and CancelRequestContextWatcherHandler, added in pgx 5.6.0, which delay the interruption. For a query still queued in PgBouncer, a cancel request is dropped. A deadline-based delay helps only if it covers the queue wait plus the query's run time. We did not run pgx for this guide.
• PgBouncer query_wait_timeout: the longest a query may wait for a database connection before PgBouncer disconnects the client. The default is 120 seconds. Because the client has no database connection yet, none is lost.
• PgBouncer checks query_wait_timeout in a maintenance loop that runs three times a second, so it cannot enforce a 50 millisecond limit. One commenter on pgx issue 1780 suggests setting it no higher than your app deadlines.
• PgBouncer min_pool_size: keeps that many connections open even when idle, if the database entry has a fixed user or a client is connected. In our lab we set it to 4 so the run started with a full pool.

How to check your own setup

Search your PgBouncer log for the reason "client disconnect while server was not ready". The same reason appears when a client leaves during an open transaction, so a few are normal. As our rule of thumb, dozens per minute during bursts point to this pattern; our collapsing run logged 225 during one 15-second test.

Then watch SHOW POOLS on the PgBouncer admin console during a busy minute. The sv_login column counts connections still logging in. If it sits at 1 while cl_waiting, the count of queued clients, keeps growing, the pool is stuck refilling.

Their startup needed a working database connection. Inngest's 23 October report says the services that accept events ran out of memory during a database connectivity problem, restarted, and could not start again while the database was unreachable.

During both windows, event submissions could fail or time out, with periods when ingestion was unavailable. The report names two resilience gaps: memory exhaustion, and a startup dependency that stopped the services recovering on their own.

The general lesson is to start without the dependency and report not ready until it is reachable. Kubernetes separates these ideas as liveness, meaning the process is alive, and readiness, meaning it can take traffic. Inngest's report does not say which exact changes it made.

The 15 October report lists several changes besides the pooler fix. They show how one outage exposed several smaller design choices.

The gateway problem is worth a second look. It came from the emergency move to the provider's poolers, which shows how a quick mitigation can add its own failure mode.

Use it as a story about feedback loops in shared resources. Interviewers want the mechanism, the loop and a fix that breaks the loop, not the timeline.

1. 1

   Draw the path

   App workers, a PgBouncer wait queue, a small pool of database connections and PostgreSQL. Say that a waiting query holds no connection.

2. 2

   Name the trigger

   Short deadlines on lookups, plus a burst at the top of the hour that roughly triples the query rate.

3. 3

   Explain the loss

   A client that gives up while queued is noticed late. PgBouncer hands it a connection, finds it gone mid-request and must close that connection.

4. 4

   Show the loop

   Connections die faster than the pool opens them, one at a time. Queues grow, more deadlines pass and more connections die.

5. 5

   Break the loop

   Let deadlines end the wait, not the query. Then add limits on queue time, alerts on not-ready closures and capacity for known bursts.