Webhook Design: Retries, Signatures and Ordering

Without a careful webhook design, updates go missing while the receiver is down, strangers can post fake updates, and a late retry can overwrite newer news. Each failure has a well-known fix.

First, the idea. Polling means asking a server again and again whether anything changed. It wastes calls and still finds changes late. A webhook flips this around: the system that has the news calls you.

Picture a common setup. A freight-booking platform lets textile sellers book space on trucks. Each consignment, one load of goods on one truck, moves through four statuses: booked, loaded, in transit and delivered. The setup is made up, but the problems are real.

A saree seller's order system needs every status change, so it can tell shops when stock will arrive. At first it polled the platform's API, the interface programs use to ask it for data, every 30 seconds. About 1,800 sellers did the same, which came to over 5 million calls a day (made up).

Almost every answer was no change. So the platform added webhooks. Each seller registers a URL, and whenever a consignment changes status, the platform sends an HTTP POST request to it with the details. Then three problems showed up in one bad week.

• Lost updates. The seller's server was down for 40 minutes during an upgrade. The platform sent 300 updates, got errors and gave up, so those updates were gone.
• Fake updates. Someone found the seller's webhook URL and posted a fake delivered message. The order system marked an invoice as due for goods still on the highway.
• Stale updates. A delivered update arrived before an older in transit update. The seller's screen then showed a delivered consignment back on the road.

The fixes are retries for lost updates, signatures for fake ones and version checks for stale ones. Retries also create a fourth problem, duplicates, and the guide fixes that too.

You need five words: webhook, endpoint, event, delivery attempt and signing secret. Each one comes with an example from the freight platform.

• Webhook: an HTTP request one system sends to another when something happens, without being asked. Example: the platform posts to the seller's server when consignment C-4471 is loaded.
• Endpoint: the URL that receives webhooks, run by the receiver. The seller registers one endpoint, and the platform stores it against the seller's account.
• Event: a record that something changed, with an id, a type and the details. Example: event evt_9f2c, of type consignment.delivered, for consignment C-4471.
• Delivery attempt: one HTTP call carrying one event. If it fails, the same event goes out again later as a new attempt, with the same event id.
• Signing secret: a long random value that only the platform and the seller know. The platform uses it to sign each call, and the seller uses it to check the signature.

Good webhook design makes sure every event eventually arrives, can be trusted and can be applied in the right order. That takes a few parts on the sender's side and a few habits on the receiver's side.

The first version was the naive one: the booking code called the seller's URL directly. When the call failed, nothing remembered the event, which is how the 300 updates vanished.

The fix starts with an outbox table. When a status changes, the same database transaction also writes an event row to the outbox. A transaction is a group of writes that all happen or none happen.

Without the outbox, the code could save the new status, crash before sending, and nobody would notice. A separate dispatcher then moves new outbox rows to a delivery queue, a waiting line of messages, and workers take and send them in turn.

The outbox has one catch. If the dispatcher crashes after queueing an event but before marking its row sent, it queues that event again on restart. The pattern's write-up on microservices.io says the relay might publish a message more than once.

The delivery log records every attempt, so support staff and sellers can see what happened and resend any event. Together, these parts give every seller four promises.

• Every status change becomes an event, because the outbox row is written in the same transaction.
• Every event is retried until it gets a 2xx answer or runs out of attempts. The one exception is 410 Gone, which stops delivery and disables the endpoint.
• Every attempt is signed with that seller's own secret.
• Every event carries an id and a version number, so the receiver can spot repeats and stale updates.

The seller's side has its own jobs: check the signature, skip repeats, ignore stale updates and answer quickly.

Webhook retries send the same event again, after a growing wait, until the receiver answers with a 2xx status code or attempts run out. A 2xx code is any HTTP status from 200 to 299, meaning OK. Any other answer, or no answer in time, counts as a failure.

Think of the optician near your house who phones when your new spectacles are ready. If you miss the call, they try again after lunch, then the next morning. After a few tries, they send a message and wait for you to come in.

• The optician is the sender, and your phone is the endpoint.
• A missed call is a failed delivery attempt.
• Calling after lunch, then the next morning, is the growing wait between attempts.
• The final message is the dead letter step: the event is parked in a store for failed events, and the seller gets an email.

Where the picture breaks: a phone call either connects or it does not. A webhook can succeed on the receiver while the sender sees a timeout, so the same news can arrive twice.

The growing wait is called exponential backoff. A little randomness, called jitter, is added to each wait so thousands of retries do not land at the same moment.

The platform copied the example schedule in the Standard Webhooks specification, version 1.0.0. It is an open, community-driven set of webhook rules, guided by a steering committee of engineers from companies such as Zapier, Twilio and Kong.

The example tries once immediately, then after 5 seconds, 5 minutes, 30 minutes, and 2, 5, 10, 14, 20 and 24 hours. That is ten attempts in all.

You can check the total yourself. The tenth attempt lands 75 hours, 35 minutes and 5 seconds after the first, which is 3.15 days. With 20% jitter on every wait, it lands between 2.52 and 3.78 days.

Real providers differ, as of 19 October 2026. Stripe retries live-mode events for up to three days with exponential backoff, and sandbox events three times over a few hours. GitHub does not redeliver failed deliveries automatically, so receivers request redelivery themselves.

Here is the platform's delivery policy, kept as a config file the workers read. The format is the platform's own, not a standard one.

webhook_delivery:
  timeout_seconds: 15          # no answer in 15 s counts as a failure
  success_codes: "200-299"     # only a 2xx stops the retries
  follow_redirects: false      # a 3xx is a failure; seller must fix the URL
  waits_between_attempts: [5s, 5m, 30m, 2h, 5h, 10h, 14h, 20h, 24h]
  jitter_percent: 20           # stretch or shrink each wait by up to 20%
  respect_retry_after: true    # honour Retry-After when an answer has one
  on_410_gone: disable_endpoint
  after_last_attempt: [dead_letter, email_seller]

The 15-second timeout is the low end of the 15 to 30 seconds the specification recommends. Redirects count as failures, as the specification suggests. That also stops a signed call being bounced to an address nobody checked, which matters for safety later.

Two more rules stop retries from hurting anyone. Each endpoint gets its own share of the workers, so one dead endpoint cannot delay healthy ones. If an endpoint fails for days, the platform disables it and emails the seller, as the specification recommends.

You handle duplicates by giving every event a stable id and having the receiver record which ids it has already processed. Retries make duplicates normal, so the receiver must make repeats harmless.

Webhook delivery is at-least-once: each event should arrive, but it may arrive more than once. Duplicates come from timeouts, from the outbox dispatcher resending after a crash, and from manual resends.

The Standard Webhooks specification says to use the webhook-id header as an idempotency key, a value that marks repeats of the same request. The id stays the same on every retry, and GitHub keeps the same X-GitHub-Delivery header on a redelivery. Stripe gives the same advice: log the event ids you processed and skip ones you have seen.

Stripe adds one wrinkle, as of 19 October 2026. In some cases it creates two separate events, with different ids, for one change. Its docs suggest matching the object's id plus the event type to catch those.

The seller's server saves each incoming event in an inbox table keyed by event id. If the id is already there, the event is a repeat. The server answers 200 so the platform stops retrying, and does nothing else.

CREATE TABLE inbox (
  event_id    TEXT PRIMARY KEY,
  body        TEXT NOT NULL,
  received_at TEXT NOT NULL
);

-- one row per event id; a repeat inserts nothing
INSERT INTO inbox (event_id, body, received_at)
VALUES ('evt_9f2c', '{"type":"consignment.delivered"}', '2026-10-19T10:00:00Z')
ON CONFLICT (event_id) DO NOTHING;

Your code checks how many rows the insert added: one means new, zero means repeat. We ran these statements on SQLite 3.45.1, and PostgreSQL documents the same ON CONFLICT clause.

How long to keep the event ids

Keep the ids at least as long as the sender can retry, plus any manual resend window. For this platform's retries with jitter, that is about four days, plus however long its own resend button allows.

The specification's own example says to save ids in Redis, an in-memory data store, for 5 minutes. That covers replays inside the five-minute signature window, explained in the next section, but not real retries.

A retry hours later carries a fresh timestamp and a fresh, valid signature, so only a stored id can catch it. Stripe also says it signs each retry with a new timestamp.

If you receive Stripe events, its Dashboard can resend events up to 15 days old and its CLI up to 30 days, as of 19 October 2026. Keep ids for 30 days there.

You verify a webhook signature by computing an HMAC of the received message with the shared secret and checking it matches the signature header. If it matches and the timestamp is recent, the call is genuine and unchanged.

HMAC, short for hash-based message authentication code, mixes a secret key with a message to produce a short fingerprint. Change one byte of the message, or use the wrong key, and the fingerprint comes out completely different.

The platform follows the Standard Webhooks layout. Each call carries three headers: webhook-id, webhook-timestamp and webhook-signature. The signed text is the id, a full stop, the timestamp in Unix seconds (seconds since 1 January 1970 UTC), a full stop, then the raw body.

The platform signs that text with HMAC-SHA256, meaning HMAC built on the SHA-256 hash function, and the seller's secret. It encodes the result as base64, which writes raw bytes as letters, digits, +, / and =. The header holds v1, a comma, then that signature.

Here is the check on the seller's side. The secret argument is the key as bytes: take the text after whsec_ and decode it from base64, as the fixed example later shows. The headers argument is a dictionary with lowercase names, and the body argument is the raw request bytes, read before any JSON parsing. A missing header or a non-numeric timestamp raises an error, which the handler turns into a rejection.

import base64, hashlib, hmac, time

TOLERANCE_SECONDS = 300  # five minutes

def is_genuine(secret: bytes, headers: dict, body: bytes) -> bool:
    msg_id = headers["webhook-id"]
    timestamp = headers["webhook-timestamp"]
    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        return False  # too old or too far ahead: maybe a replay
    signed = f"{msg_id}.{timestamp}.".encode() + body
    digest = hmac.new(secret, signed, hashlib.sha256).digest()
    expected = base64.b64encode(digest)
    for candidate in headers["webhook-signature"].split():
        version, _, value = candidate.partition(",")
        if version == "v1" and hmac.compare_digest(value.encode(), expected):
            return True
    return False

What just happened? The timestamp check rejects anything more than five minutes off, which blocks a replay: an attacker resending a real call they captured. Changing the id or timestamp to dodge that breaks the signature.

Next, hmac.compare_digest compares the two signatures. Python's documentation recommends it over a plain equality check to reduce the risk of timing attacks. In a timing attack, someone learns how much of a guess was right from how long the check took.

The code compares bytes, not text. On Python 3.13.16, comparing text raised TypeError when we put non-ASCII characters in the header. Comparing bytes simply returned False.

Finally, the loop accepts any one of several signatures separated by spaces. That matters during secret rotation, when the sender signs with both the old and new secret for a while. Stripe lets an old secret stay active for up to 24 hours.

Entries marked v1a are asymmetric signatures: the sender signs with a private key, and receivers check with a public key that cannot sign. Specification 1.0.0 prefers them, and recommends them when one side cannot control the other's security, as with sellers here. This guide uses v1 HMAC because it is simpler, faster and available everywhere. This check covers only v1.

A five-minute window needs accurate clocks on both servers. Both should sync their time with NTP (Network Time Protocol), the standard way computers agree on the time. Stripe's libraries also default to five minutes.

No, webhooks are not guaranteed to arrive in the order the events happened. Retries, parallel workers and network delays can all reorder them, so the receiver has to work out which update is newest.

That is exactly the stale update from the bad week. The in transit and delivered events for C-4471 went out a minute apart. The in transit attempt timed out, so its retry arrived after the delivered event had landed.

Stripe states that it does not guarantee event order, and tells receivers not to depend on it, as of 19 October 2026. Its snapshot events, the ones carrying the full object, record their created time in whole seconds. Two events can share a timestamp, so timestamps cannot settle the order.

Fix 1: a version number on every consignment

The platform gives each consignment a version number that goes up by one on every status change, stored in the same row. Each event carries the version, and the receiver ignores any event older than what it has stored.

The check must be one conditional update, not a read followed by a write. With a separate read, two workers can both see version 2, and the slower one then writes stale data.

UPDATE consignment
SET status = 'delivered', version = 4
WHERE id = 'C-4471' AND version < 4;

If the update changes one row, the event was newer. If it changes zero rows, the event was stale, or the consignment is not stored yet. A first event for a new consignment needs an insert instead. In our SQLite test, a read-then-write let stale version 3 overwrite version 4, while this update kept version 4.

An upsert, an insert that updates the row when it already exists, covers both cases in one statement. It changes one row for a new or newer event, and zero rows for a stale one:

INSERT INTO consignment (id, status, version)
VALUES ('C-4471', 'delivered', 4)
ON CONFLICT (id) DO UPDATE
SET status = excluded.status, version = excluded.version
WHERE consignment.version < excluded.version;

We ran it on SQLite 3.45.1: a new consignment was inserted, version 4 replaced version 2, and stale version 3 changed nothing. PostgreSQL documents the same ON CONFLICT DO UPDATE form.

Fix 2: a thin event, then fetch the latest state

A thin event carries only ids and the event type, not the full details. The receiver then calls the API for the current state, so whatever order events arrive in, the fetch returns the latest data.

The cost is one extra API call per event, and the API must handle that load. The Standard Webhooks specification describes thin and full payloads without forcing either. Stripe's dashboard labels thin events as recommended, as of 19 October 2026.

Fix 3: send in order from the sender

The platform could send one event at a time per consignment, holding the next until the last is delivered. It rejected this. One stuck event would block every later update for that consignment for up to 3.15 days, the length of the retry schedule, or 3.78 days with jitter.

Your server should check the signature, save the event, answer with a 2xx code within a few seconds, and do the real work afterwards. Slow handlers cause timeouts, and timeouts cause retries and duplicates.

GitHub asks receivers to answer with a 2xx within 10 seconds, or it treats the delivery as failed. Stripe says to return a 2xx before any complex logic, and suggests an asynchronous queue to absorb spikes. Both are as of 19 October 2026.

Here is the seller's receiver after the fixes. Notice that the 200 goes back before any consignment status changes.

Now follow two real events through it. C-4471 is at version 2, loaded, in the seller's database. The delivered event is version 4, and the delayed in transit event is version 3.

1. 1

   Read the raw body

   The delivered event arrives first. The handler reads the body bytes exactly as they came, plus the three headers.

2. 2

   Check the signature and time

   It recomputes the HMAC with the seller's secret and checks the timestamp is within five minutes. The fake delivered message from the bad week would stop here with a 401, meaning not authorised.

3. 3

   Insert into the inbox

   It inserts the event into the inbox table. The event id is the primary key, the column whose values must be unique, so any retry of this event will insert nothing.

4. 4

   Answer 200

   It answers 200 within a second. The platform marks the event delivered and stops retrying.

5. 5

   Apply in the background

   A worker reads the new row and runs the conditional update. Version 4 is higher than the stored 2, so C-4471 becomes delivered and the shop gets an alert.

6. 6

   Skip the stale update

   The delayed in transit event, version 3, arrives later. It passes the signature check and is new to the inbox, but the update changes zero rows because 3 is not higher than 4.

What just happened? The stale event, and any retry of the delivered one, got a 200, so the platform stopped sending them, and neither changed the data. We replayed this sequence on SQLite, and the row ended as delivered at version 4.

Test it against a fixed example and against the reference library, because a check that always returns False still passes a test that only tries forged calls. We ran the guide's check against the Python standardwebhooks library, version 1.1.0, on Python 3.13.16.

Start with a fixed example you can run anywhere with no installs. It signs a known id, timestamp and body with a known secret.

import base64, hashlib, hmac

secret = "whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw"
key = base64.b64decode(secret.removeprefix("whsec_"))  # decode, do not use the text
signed = b'msg_p5jXN8AQM9LWM0D4loKWxJek.1614265330.{"test": 2432232314}'
digest = hmac.new(key, signed, hashlib.sha256).digest()
print("v1," + base64.b64encode(digest).decode())

It should print v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE= on one line. The library's own sign function returns the same value for these inputs. If your output differs, check the key decoding first.

Then sign fresh messages with the library and feed them to your check. Here is what the guide's check returned for each case.

The whsec_ secret trap

Standard Webhooks secrets are shown as whsec_ followed by base64 text. The HMAC key is the decoded bytes after the prefix, not the text itself. Using the whole text as the key returned False for a genuine call in our test.

Stripe secrets also start with whsec_, but its scheme differs. In stripe-python 16.0.0, the key is the whole secret text, the signed text is the timestamp, a full stop and the body, and the signature is hex. So one verifier cannot serve both.

One more difference is undocumented as of 19 October 2026, so pin your version. stripe-python 16.0.0 rejected a timestamp 301 seconds old but accepted one an hour in the future. The guide's check, like the standardwebhooks library, rejects both directions.

You keep a webhook sender safe by sending from an isolated network, checking where every customer URL really points, and refusing redirects. Customers choose where your servers send requests, so a careless sender becomes a tool for attackers.

The attack is called server-side request forgery, or SSRF. A seller could register an address inside the platform's own network, and the delivery workers would call it from the inside.

OWASP, a non-profit that publishes security guidance, names webhooks as a typical SSRF case in its cheat sheet. The Standard Webhooks specification puts the network layer first. It suggests sending through a filtering proxy, a middle server that makes the calls and blocks internal addresses, such as Stripe's open-source smokescreen. It also suggests a subnet, a separate network segment, with no route to internal services.

• Run the delivery workers, or their proxy, in a separate subnet that cannot reach internal services.
• Resolve the hostname, then refuse any address that is not globally reachable, such as 10.0.0.1, 127.0.0.1 and 169.254.169.254. That last one is where AWS, Google Cloud and Azure servers read their own metadata.
• Connect only to the address you checked. Otherwise the name can point somewhere new between the check and the call, a trick called DNS rebinding.
• Never follow redirects, which could bounce a call to an internal address.
• Accept only HTTPS URLs, so the body and its signature travel encrypted.
• Give every endpoint its own random secret, and let sellers rotate it.

For the address check, Python's ipaddress module helps, but pick the property with care. Here is a filter we ran on Python 3.13.16.

import ipaddress

def allowed(ip: str) -> bool:
    addr = ipaddress.ip_address(ip)
    if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped:
        addr = addr.ipv4_mapped  # check ::ffff:127.0.0.1 as 127.0.0.1
    return addr.is_global and not addr.is_multicast

for ip in ["8.8.8.8", "10.0.0.1", "169.254.169.254", "100.64.0.1", "::ffff:127.0.0.1"]:
    print(ip, "allowed:", allowed(ip), "is_private:", ipaddress.ip_address(ip).is_private)

It prints allowed True only for 8.8.8.8. Look at 100.64.0.1, from the shared address space that carrier networks use: is_private is False, so a filter built on is_private lets it through.

is_global is also False for that address, so this filter blocks it. The mapped form ::ffff:127.0.0.1 is checked as 127.0.0.1 and blocked too. Python 3.13 changed results for some other ranges, such as 192.0.0.0/24, so pin your version and keep a test like this. The filter backs up the proxy and subnet; it does not replace them. On Python 3.13.16 it still allowed two IPv6 forms that embed 127.0.0.1: 64:ff9b::7f00:1, used by IPv6-to-IPv4 translators, and the deprecated ::127.0.0.1.

Explain it as a reliable sender built on an outbox and a queue, plus a careful receiver. Tie each part to the failure it prevents.

1. 1

   Start with polling versus webhooks

   Say why webhooks exist: fewer wasted calls and faster news. Then name the cost: you now owe customers reliable, trustworthy delivery.

2. 2

   Draw the sender

   An outbox written in the same transaction as the change, a dispatcher, a delivery queue, workers, a delivery log and a dead letter store.

3. 3

   Explain retries

   A 15 to 30 second timeout, 2xx as the only success, exponential backoff with jitter over about three days, then a dead letter and an email.

4. 4

   Add signatures

   HMAC-SHA256 over the id, timestamp and raw body, a five-minute window, a constant-time comparison, and two signatures in the header during rotation. Mention that the specification prefers public-key v1a signatures when you cannot secure both sides.

5. 5

   Handle duplicates and order

   At-least-once delivery, a unique inbox keyed by event id and kept longer than the retry window, and a version checked in one conditional update.

6. 6

   Finish with safety and operations

   An isolated subnet or filtering proxy for outgoing calls, a resolved-address check, a fair share of workers per endpoint, a resend button and an events API.