Idempotency Keys Explained: Safe Retries Without Double Charges

If the first request already worked and only its reply got lost, a plain retry charges the customer twice. The app sees a timeout, sends the same payment again, and the bank debits again. An idempotency key is the standard fix.

Picture a common setup. A bill-payments app in India lets people pay electricity, water and mobile bills through UPI, India's instant bank-to-bank payment system. The setup is made up, but the problem is real.

On the last evening before electricity bills were due, the app's partner bank slowed down. Replies that usually took one second took twenty. The app gave up waiting after 15 seconds and quietly tried again.

Most of those first requests had not failed. The bank had debited the customer and was still sending its reply when the app stopped listening. The retry looked like a brand new payment, so the bank debited again.

By morning, 386 customers had paid one bill twice, about Rs 4.1 lakh (410,000 rupees) in extra debits. Support got over 600 complaints in two days. One shopkeeper paid Rs 2,300 twice and was short of cash for stock that week.

The first idea in the room is usually to switch retries off. That is the wrong fix. Phones on weak networks need retries. The real bug was retrying without any way for the server to recognise a repeat.

A request can fail at three points, and after a timeout the phone cannot tell them apart. Stripe, a payments company, listed them in a 2017 engineering blog post on idempotency.

• The request never reaches the server. A retry is safe, because nothing happened.
• The server fails halfway through the work. A retry may or may not be safe, depending on how far it got.
• The server finishes, but the reply is lost on the way back. A plain retry repeats the work, which is exactly what hit the bill-payments app.

After a timeout, all three look the same from the phone: no reply. In the diagram, the payments API is the set of addresses the app calls to ask its server to take a payment. Follow the diagram below from the tap to the complaint. Each box is one message, and nothing in the path can tell the second request is a retry.

The fix is an idempotency key, a small ID that turns a risky retry into a safe one. Pause and predict before reading on: when the same key arrives twice, should the server refuse the second request, or answer it?

An idempotency key is a unique ID that a client attaches to a request that changes something, so the server can tell a retry from a new request. The server does the work once per key and replays the saved result for every repeat.

So the answer to the prediction is: answer it, with the same reply as the first time. The client then learns what really happened, and nothing happens twice.

The key usually travels in an HTTP header called Idempotency-Key. HTTP is the protocol apps use to talk to web servers. A header is a labelled line of extra information sent along with a request.

The value is a long random ID, such as a UUID (universally unique identifier). A version 4 UUID, the random kind, has 122 random bits, so two phones are practically never going to make the same one by chance.

Think of a chequebook. Every leaf has its own printed number. When the bank pays a cheque, it marks that number as paid, and it will not pay the same number twice.

An idempotency key is that printed number. The payment details are what you write on the leaf. The server plays the bank: it honours each number once.

Where the picture breaks: a bank refuses a cheque number it has already paid, but a server does not refuse a repeated key. It sends back the reply it gave the first time. A bank also keeps records for years, while a server forgets keys after a set time.

An API call is idempotent when sending it many times has the same effect on the server as sending it once. The HTTP standard, RFC 9110, defines GET, HEAD, OPTIONS, TRACE, PUT and DELETE as idempotent. POST and PATCH are not guaranteed to be.

Those names are HTTP methods, the verb at the start of every request. GET reads, POST creates, PUT replaces a whole record, PATCH changes part of one, and DELETE removes one.

Here is what that looks like in the bill-payments app. A GET that reads a bill changes nothing, so asking ten times is harmless. A PUT that sets your phone number leaves the same number however often it runs.

A DELETE that removes a saved biller finds nothing left on the second try. Notice that idempotent is about the effect, not the reply. The second DELETE may get a different reply, such as the status code 404 Not Found, yet the server ends in the same state. A status code is the three-digit number at the start of every HTTP reply.

POST is not on the list. An endpoint is one address on the API that does one job. A POST to the payments endpoint means create a new payment, so two POSTs mean two payments.

That is why a client that gets no reply to a POST cannot know whether a retry is safe. The IETF, the group that writes internet standards, has a draft for an Idempotency-Key header to fix exactly this, for methods such as POST and PATCH.

The idea is still common in production. The draft's implementation list names 19 APIs and frameworks that use this header or a similar mechanism, many of them payment companies. Stripe's API v1, for example, accepts an idempotency key on every POST request.

The client creates one key per action and sends it with every attempt. The server claims the key before doing any work, saves the result afterwards, and answers any repeat from what it saved.

1. 1

   Customer taps Pay

   The app creates a new key for this payment, saves it on the phone, and sends it with the request for Rs 2,300.

2. 2

   Server claims the key

   The payments API inserts the key into a table that allows each key only once. If the insert works, this key is new.

3. 3

   Server does the work

   It asks the bank to debit the customer and records the payment. Then it saves the response, such as the payment ID and status, next to the key.

4. 4

   The reply is lost

   The bank was slow and the app timed out. The money moved, but the phone does not know that.

5. 5

   App retries with the same key

   The app sends the identical request with the identical key. It must never make a new key for a retry.

6. 6

   Server replays the result

   This time the insert finds the key already there, marked done. The server returns the saved response, and no second debit happens.

Here is the server's decision path. Start at the incoming request and answer each question in turn. Notice that only one branch ever reaches the bank.

The server should return the saved result of the first request and never redo the work. The exceptions are a key still in progress, a key reused for a different request, and a key that has expired.

To spot a reused key, the server stores a fingerprint with it. A fingerprint is a hash of the request body: a short, fixed-length code that changes if any detail changes. The same key with a different fingerprint means the client made a mistake.

The IETF draft makes the fingerprint optional. Without it, or a stored copy of the body, the server cannot notice that a key came back with a different amount.

Back to the chequebook. If a cheque came back with a number already paid but a different amount written on it, the bank would stop and ask questions. The fingerprint is how the server notices the changed amount.

The table lists every case the payments API handles. The 400, 409 and 422 replies below are the ones the IETF draft suggests.

The draft says the server should respond with the result of the earlier operation, success or error. Stripe's API v1 works the same way. It saves the status code and body of the first request, including a 500 error, which means the server itself failed.

Stripe's v1 docs add one subtle rule. A result is saved only after the endpoint starts real work. If a request fails input checks, or clashes with one already running, nothing is saved, so the client can retry.

Stripe's newer API v2 behaves differently. On a retry it tries to finish failed or half-done work, instead of replaying the error. So check which behaviour your provider documents before you copy it.

The client should create the key, because only the client knows whether two requests are one action or two. The server cannot tell two real Rs 2,300 payments from one payment sent twice.

That second case is common. A shopkeeper may pay two different Rs 2,300 bills on purpose in one evening, one for the shop and one for home. Only the app knows that each tap was a new action.

The payments team wrote these rules for the app:

• Make one key per user action, at the moment the action starts. One tap on Pay is one key.
• Save the key on the phone before the first send, so a crash or restart still reuses it.
• Reuse the exact same key and the exact same body for every retry of that action.
• Use a random ID such as a version 4 UUID. The IETF draft recommends a UUID or a similar random identifier, and Stripe suggests version 4 UUIDs.
• Never put personal data in the key. Stripe's docs warn against sensitive data such as email addresses.
• Make a new key only when the user starts a new action, such as after editing the amount.

The team also scopes keys per user. The unique rule in the key table covers the user ID and the key together, not the key alone. The IETF draft warns that with easy-to-guess keys, an attacker could fetch other clients' saved replies.

You need a table with a unique rule on the user and key, an insert that claims the key before any work, and an update that saves the response after. The database does the hard part.

CREATE TABLE idem_keys (
    user_id    bigint      NOT NULL,
    key        text        NOT NULL,
    fp         text        NOT NULL,
    status     text        NOT NULL,
    code       int,
    response   jsonb,
    created_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (user_id, key)
);

The primary key on user_id and key is the unique rule. It also scopes each key to one user, as the previous section suggested.

Here is the handler, trimmed to the core. The db argument is a connection from psycopg 3, a widely used Python driver for PostgreSQL. It is opened with autocommit=True, so each statement is saved at once, and with namedtuple_row, so rows have named fields.

The bank_debit function stands in for the call to the partner bank. It returns a status code and a response.

import hashlib, json

def pay_bill(db, user_id, key, body):
    fp = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
    claimed = db.execute(
        "INSERT INTO idem_keys (user_id, key, fp, status) VALUES (%s, %s, %s, 'started') "
        "ON CONFLICT DO NOTHING RETURNING key", (user_id, key, fp)).fetchone()
    if claimed is None:  # this user has sent this key before
        row = db.execute("SELECT fp, status, code, response FROM idem_keys "
                         "WHERE user_id = %s AND key = %s", (user_id, key)).fetchone()
        if row.fp != fp:
            return 422, {"error": "key already used for a different payment"}
        if row.status == "started":
            return 409, {"error": "still processing, retry shortly"}
        return row.code, row.response  # replay: no second debit
    code, response = bank_debit(body, reference=f"pay-{user_id}-{key}")
    db.execute("UPDATE idem_keys SET status = 'done', code = %s, response = %s "
               "WHERE user_id = %s AND key = %s", (code, json.dumps(response), user_id, key))
    return code, response

The INSERT with ON CONFLICT DO NOTHING is the claim. PostgreSQL adds the row only if this user and key pair is new; otherwise it quietly does nothing.

RETURNING gives a row back only when a row was really inserted. So claimed is None means the key was already there. The code then runs the replay checks and never reaches the bank.

We ran this exact handler on PostgreSQL 16.15 with psycopg 3.3.6 and Python 3.13. A retry with the same key and body got the saved reply, a changed amount got 422, and a bank decline was replayed too. The bank was called once per key.

What happens when two retries arrive at once

Only one insert can win the unique rule. With autocommit, the loser's insert waits at most until the winner's insert is saved, then does nothing. It then finds status started and gets a 409, or the saved reply if the first attempt already finished.

In our test, two threads sent the same key at the same moment, 1,000 times. The bank was called exactly once in every round. The second thread almost always got a 409. In a handful of rounds the first had already finished, so it got the saved reply; that split changes from run to run.

Do not wrap the claim and the bank call in one open transaction. In our test, the second request then sat blocked at its insert until the first one committed.

Worse, a crash after the debit rolls back the claim but not the money. In our test the claim row vanished, so the retry claimed the key again and the bank was called twice.

Pass the key on to the bank

The bank call passes a reference built from the key. If the bank also rejects repeats of the same reference, even a retry from the app's own server cannot debit twice. Ask your bank or gateway whether it does this.

Check the claim step yourself

You can watch the claim work in psql, PostgreSQL's command-line client. Create the table above, then run this insert twice. The last number in INSERT 0 1 is the count of rows inserted.

INSERT INTO idem_keys (user_id, key, fp, status)
VALUES (42, 'demo-key-1', 'abc', 'started')
ON CONFLICT DO NOTHING RETURNING key;

-- first run prints:   demo-key-1   (1 row)   INSERT 0 1
-- second run prints:  (0 rows)               INSERT 0 0

The team keeps the policy in a small config file, so support and security teams can read it without opening code.

idempotency:
  header: Idempotency-Key
  required_for:
    - pay_bill
    - refund_payment
  key_format: uuid_v4
  unique_per: user
  keep_keys_for_hours: 48
  stuck_after_minutes: 5
  when_stuck: ask_bank_for_status

required_for lists the calls that reject a request without a key, with a 400. keep_keys_for_hours is the expiry policy the IETF draft asks each API to publish. The last two lines handle a crash, explained further down.

Keep each key longer than the longest time a client might still retry, and tell clients that time. Stripe's API v1, for example, may remove keys once they are at least 24 hours old, and treats a later reuse as a new request.

The bill-payments app keeps keys for 48 hours. The app stops retrying a payment after 10 minutes and shows a status screen instead. That leaves a wide gap between the last possible retry and the expiry.

The cost is storage. At 200,000 payments a day, 48 hours means about 400,000 rows. In our test, 400,000 rows with a UUID key and a 240-byte saved reply took about 205 MB in PostgreSQL 16, including the index. That was a freshly loaded table; updates and deletes leave dead rows behind, so expect a live table to be somewhat larger.

That is small for any database. A cleanup worker deletes expired rows every hour. To measure your own table, run SELECT pg_size_pretty(pg_total_relation_size('idem_keys')); in psql. For our test table it printed 196 MB, because PostgreSQL counts a MB as 1,048,576 bytes.

Idempotency keys stop duplicate requests, but three problems remain. A crash or a bank timeout can leave the work half done, a payment can be saved apart from its key, and clients can break the rules.

A crash or bank timeout in the middle of a payment

Suppose the bank debits the customer, then the app's server crashes before it saves the response. The key is stuck in started. Every retry gets a 409, and nobody knows whether money moved. The same happens without any crash if the bank call times out or raises an error, because the handler never reaches its update.

The fix is the stuck_after_minutes rule. A worker finds keys left in started for over five minutes and asks the bank for the payment's status by its reference. Then it saves the true result. Matching your records against the bank's like this is called reconciliation.

Here is the full setup. Look at the two workers: one rescues stuck keys, and one clears old ones.

A payment saved apart from its key

If the payment record and the key's status live in the same database, update them in one transaction after the bank replies. A transaction is a group of writes that all happen or none do. Then you never get a payment without a finished key, or the reverse.

Clients that break the rules

• A new key on every retry. The server sees new actions and debits again, so test the app's retry code, not just the server.
• A changed body on retry, such as a fresh timestamp field. The fingerprint differs and the server returns 422, so keep retries identical.
• Retrying with no pause. Keys stop duplicates, not overload, so pair them with backoff and jitter, which spread retries out over time.
• Keys on the first hop only. If your server calls the bank, that call needs its own protection, like the reference in the handler above.

Explain the lost-reply problem first, then the key, then the server's states and the edge cases. Use one concrete payment as your example all the way through.

1. 1

   Show why retries are risky

   Draw the timeout case: the bank debited, the reply was lost, and the retry looks new. Say that after a timeout the client cannot tell the three failure points apart.

2. 2

   Add the key

   The client makes one random key per action and sends it in an Idempotency-Key header on every attempt.

3. 3

   Claim before work

   The server inserts the key under a unique rule before calling anything. Only the winner of that insert does the work.

4. 4

   Save and replay

   Store the fingerprint, status and response with the key. Repeats get the saved response, a 409 while running, or a 422 for a changed body.

5. 5

   Cover the edges

   Mention expiry, per-user scope, a checker for stuck keys, and passing a reference on to the bank.

6. 6

   Name the limits

   Keys stop duplicate requests. They do not replace backoff, rate limits or reconciliation with the bank.