Agent Failure Recovery: Checkpoints and Partial Failures

Without agent failure recovery, the work is either lost or done twice. Steps that finished before the crash run again, and a step that half-finished leaves a mess nobody can see.

Picture a common setup. A wholesale distributor in Ludhiana runs an AI agent every night that checks supplier invoices against the stock that actually reached the warehouse. The setup is made up, but the problem is real.

An AI agent here means a program where a model (the AI that reads and writes text) helps decide the next step, and your code runs tools. A tool is an action the code allows, such as reading the stock table.

The nightly run has ten steps. The order matters later, so keep it in mind.

• Steps 1 to 3: fetch the day's invoices, have the model pull out the line items, and match each line to a purchase order.
• Steps 4 to 6: read the stock received, compare quantities and prices, and flag every mismatch.
• Step 7: post adjustment entries to the accounts ledger, one per mismatch.
• Step 8: update the stock records.
• Step 9: email a debit note (a formal claim for money back) to each supplier who billed too much.
• Step 10: send the finance head a summary.

One Tuesday night the run has 312 invoices and 40 adjustments to post. At entry 24 of step 7, the cloud machine running the agent is shut down for maintenance. The process dies without a word.

The first version had no recovery at all. A scheduled job noticed the run had stopped and started it again from step 1. It read all 312 invoices again, paying the model a second time, and then posted all 40 entries.

The next morning the accountant found 23 adjustments posted twice. The ledger was off by Rs 2,71,500 (about 2.7 lakh, or 271,500 rupees). She spent the day reversing entries by hand, and the team asked why a restart repeats finished work.

In the diagram below, look at where the crash landed. Every step before it changed nothing outside the agent. The ledger box is the first step that touched someone else's system.

Agent failure recovery is the set of rules that lets a stopped agent finish its task correctly. It continues from where it stopped, keeps finished work, and does no outside action twice.

Two words come up all the time. A crash is when the process stops suddenly: the machine restarts, memory runs out, or someone deploys new code. A partial failure is when some effects of a step happened and others did not.

Good recovery answers four questions. Each one gets its own section below.

• Where are we? A checkpoint records which steps finished and what they produced.
• Did the last action happen? Idempotency keys make it safe to try again when you cannot tell.
• Can we undo it? Compensation steps reverse work that should not stand.
• Who decides when we cannot? A clear handover to a person, with the evidence attached.

Think of a long Himalayan trek with camps along the way. If snow closes the path on summit morning, nobody walks back to the first village to start again. The group waits at the last camp and tries the next day.

• The camp is the checkpoint: a safe place where progress is kept.
• A supply drop labelled with the group's name is the idempotency key. Before carrying up more, the guide checks whether that labelled drop is already there.
• Turning back below the summit is the handover. Stopping safely is the plan working, not the plan failing.

Where the picture breaks: a trek group can see which camp it reached. A crashed agent cannot see whether its last call landed, so it must ask the other system.

A checkpoint is a saved record of an agent's progress, written to durable storage after each step. Durable storage is storage that survives a crash, such as a database table or a file on disk.

Keeping progress only in the program's memory was the first version's mistake. Memory disappears when the process dies. A checkpoint lives outside the process, so a new process can read it.

The team's checkpoint for one nightly run holds five things.

• The run id, such as run-2026-10-13-ldh, built from the date and the warehouse, so every record belongs to exactly one run.
• The list of steps that are done, in order.
• The output of each finished step: invoice lines, matches, the stock numbers used and the mismatch list.
• Every model answer the run used, saved word for word.
• A count of retries per step, so a broken step cannot loop forever.

Saving the stock numbers matters more than it looks. If the run resumes at 6 a.m., the warehouse may have received new goods overnight. Reading stock again would compare old invoices against new counts and flag false mismatches.

Saving model answers matters for the same reason. Asking the model again costs money, and the new answer may differ. Then the new invoice lines might not match the entries already posted from the first answer.

Big systems use the same idea. Apache Flink, a stream processing engine, checkpoints its state and its position in each input stream. Its documentation says this gives the application the same semantics as a run without failures.

That promise has conditions. Flink needs a source that can replay records. Exactly-once results in outside systems also need the sink, the part that writes output, to join checkpointing or make writes safe to repeat.

You give every outside action an idempotency key, a unique name for that one action, and the receiving system ignores a second request with the same key. Then repeating a step is harmless.

A checkpoint is saved only after a whole step ends. Tuesday's crash landed inside step 7, after entry 23. Even with checkpoints, the resumed run sees step 7 as not done and starts it again from entry 1.

The team builds each key from the run id, the invoice number, the line number and the mismatch kind, such as price or quantity. The same mismatch always gets the same key, and two different mismatches never share one.

A key must name exactly one action. If two different actions share a key, the ledger silently skips the second one.

The key must be stored by the ledger, the one place that really knows what was posted. A key kept only in the agent's memory would die with the crash.

The IETF, the group that writes internet standards, drafted an Idempotency-Key HTTP header for this. It aims to make requests like POST fault-tolerant, since a repeated POST may take effect twice. Its implementation list includes several payment APIs.

The latest version, draft 07 of 15 October 2025, expired on 18 April 2026 without becoming a standard, as of 19 October 2026. The idea is still in wide use; only the shared header is not standardised.

A partial failure is when a step or a run stops after some of its effects happened and others did not. The danger is that the agent often cannot tell which part happened.

Tuesday's crash was a partial failure inside one step: 23 entries posted, 17 not. A timeout is the sneakiest kind. The request reached the ledger and the entry was saved, but the reply got lost.

The right recovery depends on what kind of step failed. The team sorted the ten steps into five kinds and wrote one rule for each.

The outbox in the table is a small table where the agent writes each email it wants to send, along with its key. A separate sender reads the outbox, sends each message and marks it sent.

Not every error deserves a retry, either. The team splits errors into three groups, because each one needs a different response.

• Temporary: a timeout or a busy server. Try at most three times, waiting longer between tries, with the same key.
• Permanent for one item: an invoice scan nobody can read. Set that invoice aside, note why, and finish the others.
• Unknown or growing: the ledger rejects every entry. Stop the run and hand it to a person.

Some failures raise no error at all. A 2025 study of multi-agent systems, where several agents work together, sorted failures from real runs into 14 modes in three groups.

One group is task verification: ending a task too early, or checking the work badly or not at all. So mark a step done only after checking its result, such as the ledger holding all 40 keys.

You run a compensation step for each finished step: a new action that cancels the effect of the old one. You do not delete history; you add a correcting entry on top of it.

Sometimes a finished run should not stand. Say a supplier calls the next morning: its prices changed by agreement last week, so its 6 mismatches were wrong. Those 6 entries, their stock changes and one debit note need undoing.

The idea is old. In 1987, Hector Garcia-Molina and Kenneth Salem described sagas: a long-lived transaction written as a sequence of smaller transactions. Either all of them complete, or compensating transactions run to amend the partial work.

• Ledger entry: post a reversal entry with its own key, pointing to the original.
• Stock update: post the opposite adjustment, with a note on why.
• Debit note: you cannot unsend an email, so send a withdrawal note that cancels the claim.
• Read-only and model steps: nothing to undo, because they changed nothing outside.

Run compensations newest first, so each undo starts from the state its own step left. They also need idempotency keys, because the undo itself can crash halfway. Record each one in the same checkpoint, so an interrupted undo also resumes.

The order of the steps helps too. The run sends debit notes in step 9, after the ledger and stock updates. The step that is hardest to undo runs only once everything before it has worked.

An agent should stop and hand over when its retries are used up or the error is unknown. It should also stop before a step that cannot be undone when the evidence is unclear.

Follow the questions in this diagram from the top. It is the check the agent runs every time a step fails. Only one path leads to an automatic retry.

A good handover note saves the person from digging. It names the run, the failed step, the exact error, what was already posted, and what the agent would have done next.

The person then picks one of three actions: resume, resume after a fix, or undo the run. All three use the same checkpoint, so nothing is lost while the run waits.

You restart the run with the same run id, load its checkpoint, skip every step marked done, and continue from the first unfinished step. Earlier results come from the checkpoint, not from running those steps again.

Here is the core of the runner, tested on Python 3.13. step_funcs is a dictionary that maps each step name to the function that does it. Each function gets the run id and all earlier results.

import json
import os
import pathlib

STEPS = ["fetch_invoices", "extract_lines", "match_orders", "read_stock",
         "compare", "flag_mismatches", "post_ledger", "update_stock",
         "send_debit_notes", "send_report"]

def save_checkpoint(path, state):
    tmp = path.with_suffix(".tmp")
    with open(tmp, "w") as f:
        json.dump(state, f)
        f.flush()
        os.fsync(f.fileno())          # push the bytes to the disk first
    os.replace(tmp, path)             # then swap the new file in, in one step
    folder = os.open(path.parent, os.O_RDONLY)
    os.fsync(folder)                  # make the swap itself survive a power cut
    os.close(folder)

def run(run_id, step_funcs):
    path = pathlib.Path(f"checkpoints/{run_id}.json")
    path.parent.mkdir(exist_ok=True)
    state = json.loads(path.read_text()) if path.exists() else {"done": [], "data": {}}
    for step in STEPS:
        if step in state["done"]:
            continue                  # finished before the crash
        state["data"][step] = step_funcs[step](run_id, state["data"])
        state["done"].append(step)
        save_checkpoint(path, state)
    return state["data"]["send_report"]

Two parts carry the weight. The continue line is the skip: a resumed run jumps past steps 1 to 6 and reuses their saved output. On Tuesday's run, that alone would have saved the second model bill.

save_checkpoint writes a temporary file, forces it to disk with fsync, then swaps it in with os.replace. On POSIX systems such as Linux and macOS, that rename is atomic: readers see the old checkpoint or the new one, never half.

On Linux, fsync on the file does not promise that the rename itself reached the disk. So the last three lines of save_checkpoint fsync the folder too, as the fsync manual page advises.

Without them, a power cut could bring back an older checkpoint, and finished steps would run again, which may include a paid model call. In production the team keeps checkpoints in a database table instead, where one commit covers both concerns.

How the parts fit together

A small supervisor watches each run's heartbeat, a timestamp the worker updates every 30 seconds (made up). If a heartbeat goes quiet for two minutes, the supervisor gives the run to a fresh worker.

In the diagram below, notice that the worker never talks to the supervisor directly. Both only read and write the checkpoint store. That way, a dead worker cannot take anything that matters with it.

How workflow engines do the same job

Durable workflow engines build this in. Temporal, an open-source workflow engine, keeps an Event History: its documentation calls it a complete, ordered log of everything that has already happened in a workflow.

After a crash, Temporal starts the workflow code from the beginning and replays that history step by step to rebuild its state. So the code must make the same decisions given the same history.

Values that differ between runs, like the current time or a network reply, must come from the history. That is the same rule as saving model answers in your own checkpoint.

Even then, Temporal's documentation recommends that Activities, the steps that call outside systems, be idempotent. A failed Activity attempt is retried automatically, so an engine does not remove the need for keys.

Let the system that stores the write refuse a second copy of the same key. A unique key enforced by the ledger works even when your own checks are wrong or two workers race.

In this version, a SQLite table stands in for the real ledger. The reference column is the primary key, so the database itself can hold only one row per key.

import sqlite3

db = sqlite3.connect("ledger.db")
db.execute("CREATE TABLE IF NOT EXISTS entries ("
           "reference TEXT PRIMARY KEY, invoice_no TEXT NOT NULL, amount INTEGER)")

def post_ledger(run_id, data):
    posted = []
    for entry in data["flag_mismatches"]["adjustments"]:
        key = f"{run_id}:{entry['invoice_no']}:{entry['line']}:{entry['kind']}"
        with db:                      # one transaction per entry
            db.execute("INSERT INTO entries VALUES (?, ?, ?) "
                       "ON CONFLICT(reference) DO NOTHING",
                       (key, entry["invoice_no"], entry["amount"]))
        posted.append(key)
    return {"posted": posted}

ON CONFLICT(reference) DO NOTHING skips the insert when that key already exists. It needs SQLite 3.24 or newer; our Python 3.13 test ran on SQLite 3.45.1. Python prints yours with sqlite3.sqlite_version. Other errors still raise: invoice_no is NOT NULL, so a missing invoice number fails loudly.

Each entry is its own transaction, so a crash keeps the entries already committed. The resumed step walks all 40 again, and the first 23 are quietly skipped.

One case DO NOTHING hides: a retry that reuses a key with a different amount. Treat that as a bug, not a duplicate, by comparing the stored amount. Under the IETF draft, a server should answer such a request with HTTP status 422, an error code.

Why check-then-post is weaker

A simpler version asks the ledger whether the key exists, then posts if not. That is safe only while exactly one worker runs the step, because two workers can both check before either posts.

In our test, two threads ran check-then-post on the same 40 entries, with a 2 ms pause between check and post. 14 of 15 tries left duplicates, up to 44 rows. With the primary key, all 15 left exactly 40.

So if your ledger is an outside API, look for an idempotency key or a reference field it enforces. If it only lets you look entries up, keep check-then-post and keep one worker per run.

Why a lease alone does not stop two workers

A lease is a lock that expires unless renewed, so only one worker owns a run at a time. It cuts the risk of two workers but does not remove it.

Martin Kleppmann's well-known post on distributed locking shows the gap. A worker can freeze, for example in a long garbage-collection pause, when the runtime stops the program to free memory. Its lease expires and another worker takes over. When it wakes, it still writes.

His fix is a fencing token: a number that grows each time the lease changes hands. The storage remembers the highest number it has seen and refuses writes with an older one.

Why an outbox can still send twice

An outbox gives at-least-once delivery, not exactly-once delivery. If the sender crashes after sending an email but before marking it sent, it sends that email again on restart.

The usual fix is on the receiving side: it tracks message ids and ignores repeats. A supplier's inbox cannot do that, so pass the outbox key to your email service if it accepts one.

If it does not, a rare duplicate email is the price. So write each debit note to say which claim it is, so a repeat is easy to spot and ignore.

Kill the agent inside a step that writes, run it again, and count the rows. The demo below wires stand-in steps to the runner and ledger above and dies at entry 24 of step 7.

Save the runner and the ledger code together as recovery.py. Save this file next to it as demo.py.

Pause and predict: the crash leaves 23 rows. How many rows will the ledger hold after the resumed run posts all 40 entries again?

import os
import sqlite3
import sys
from recovery import STEPS, run, post_ledger

adjustments = [{"invoice_no": f"INV-{n}", "line": 1, "kind": "qty", "amount": 500}
               for n in range(1, 41)]

def stub(name, result=None):
    def step(run_id, data):
        print("ran", name)
        return result
    return step

def post_then_crash(run_id, data):
    post_ledger(run_id, {"flag_mismatches": {"adjustments": adjustments[:23]}})
    os._exit(1)                       # the machine dies inside step 7

def post_and_say(run_id, data):
    print("ran post_ledger")
    return post_ledger(run_id, data)

funcs = {name: stub(name) for name in STEPS}
funcs["flag_mismatches"] = stub("flag_mismatches", {"adjustments": adjustments})
funcs["post_ledger"] = post_then_crash if "--crash" in sys.argv else post_and_say

run("run-demo", funcs)
rows, keys = sqlite3.connect("ledger.db").execute(
    "SELECT COUNT(*), COUNT(DISTINCT reference) FROM entries").fetchone()
print(f"ledger rows: {rows}, distinct keys: {keys}")

Run python3 demo.py --crash first. It prints ran for steps 1 to 6, then dies with exit code 1, leaving 23 rows in ledger.db and six steps in the checkpoint.

Now run python3 demo.py. Only steps 7 to 10 print ran, and the last line reads ledger rows: 40, distinct keys: 40. A third run prints only that last line, because every step is done.

Compare that with the first version. Our test of a restart from step 1 with plain inserts left 63 rows: 40 entries plus 23 repeats, and 624 invoice reads instead of 312.

Here is how the next real crash went, three weeks after the changes. A memory error killed the worker at entry 31 of 40 in step 7.

1. 1

   The supervisor notices

   The run's heartbeat stops. Two minutes later, the supervisor sees the lease has expired and hands the run to a fresh worker.

2. 2

   The worker loads the checkpoint

   Steps 1 to 6 are marked done. The saved invoice lines, stock numbers and mismatches load from the store, with no model calls.

3. 3

   Step 7 walks every entry again

   For entries 1 to 30, the ledger already holds the key, so the inserts are skipped. Entry 31 is tried too, because the crash may have hit after it committed.

4. 4

   The rest is posted once

   Entry 31 was not in the ledger, so it and entries 32 to 40 are posted. Step 7 is marked done in the checkpoint.

5. 5

   The run finishes normally

   Stock is updated, the outbox sender sends each debit note, and the summary notes one restart at step 7, with no duplicate entries.

What just happened? The checkpoint saved steps 1 to 6 and the model bill. The ledger's keys made step 7 safe to repeat. The accountant noticed nothing, which is what good recovery looks like from outside.

Explain it as three layers: checkpoints for progress, idempotency for repeated actions, and compensation plus human handover for work that must be reversed or judged.

1. 1

   Name the failure first

   Say what can crash and where: a worker dies mid-step, a call times out, or a model answer turns out wrong.

2. 2

   Sort the steps by kind

   Mark each step as read only, model call, business write, outside message or wait for a person. Recovery rules follow from that list.

3. 3

   Add checkpoints

   Save done steps, outputs and model answers to durable storage after each costly step. Resume with the same run id.

4. 4

   Make writes safe to repeat

   Give each outside action an idempotency key built from stable ids, enforced by the system that stores it. Put the irreversible steps last.

5. 5

   Plan the undo and the handover

   List a compensation for every write. Cap retries, park bad items, and hand stuck runs to a person with full context.

6. 6

   Show how you detect a crash

   Mention heartbeats, leases with fencing tokens so a stale worker cannot write, and metrics such as restarts per night and parked items.