Delivery Guarantees: At-Most, At-Least and Exactly-Once

The shopper gets the points twice, and the company pays cashback nobody earned. It usually happens when a worker adds the points, crashes before saving its progress, and handles the same purchase again after restarting. Stopping this is what exactly once semantics is about.

Picture a common setup. A loyalty service runs cashback points for about 1,200 shops in Chandigarh. Shoppers earn 1 point for every Rs 10 they spend, and 100 points turn into Rs 10 of cashback. The setup is made up, but the problem is real.

Every bill paid at a shop counter becomes a purchase event, a small record of what happened. About 80,000 of them pass through a message queue each day. A message queue is a service that holds messages until a program reads them.

A program called the points worker reads each purchase event and adds points to the shopper's wallet. One Saturday evening, the team deploys a new version of it. The old copies stop halfway through a batch.

When the new copies start, they carry on from the last saved position. But the old copies had already added points for many events after that position. Those purchases get points a second time.

On Monday, the cashback report shows the damage. About 3,400 purchases were credited twice, which is 2.1 lakh (210,000) extra points. That is Rs 21,000 of cashback nobody earned.

The payments lead asks the team to make every purchase count exactly once. The honest answer is yes, but not in the way most people imagine. The rest of this guide explains that answer and the fix the team ships.

A delivery guarantee is the promise a messaging system makes about how many times each message reaches the program that reads it. There are three: at most once, at least once and exactly once.

The names are old and widely shared. MQTT, an open messaging standard used by small devices, names its three quality of service levels at most once, at least once and exactly once delivery.

You need a few more words before going further. Each one gets a plain meaning and its place in the loyalty setup.

Producer and consumer. A producer is the program that sends messages, and a consumer is the program that reads them. Here the payments service is the producer and the points worker is a consumer.

Broker, topic and partition. A broker is the server that stores messages between the two. The team uses Kafka, an open-source system that keeps messages on disk. Messages sit in a topic, a named stream such as purchases.

Kafka splits each topic into partitions, separate ordered lists of messages that different brokers can store. Order holds inside one partition, not across the whole topic.

Acknowledgement. An acknowledgement, or ack, is a short reply that means got it. The payments service waits for Kafka's ack after each send. If none arrives in time, it sends again.

Offset. An offset is a message's position number inside a partition. A consumer saves the offset it has finished, so after a restart it knows where to carry on. Saving it is called committing the offset.

Here is the pipeline. Follow one purchase from the shop counter to the shopper's wallet and phone. Every arrow is a place where a message can be lost or repeated.

At-most-once delivery may lose a message but never repeats it, while at-least-once delivery never loses a message but may repeat it. For a consumer, the choice comes down to when it saves its progress.

Look at the points worker. For each event it does two things: it adds points, and it commits the offset. The order of those two steps decides which guarantee you get.

At most once: save progress first

If the worker commits the offset first and adds points second, a crash in between loses that purchase. On restart, it carries on after the saved offset. The shopper never gets those points.

At-most-once suits data where a small loss hurts nobody. The loyalty app also sends tap events for a usage dashboard. Losing a few taps out of millions changes no decision anyone makes.

At least once: do the work first

If the worker adds points first and commits the offset second, a crash in between repeats that purchase. On restart, it reads the event again and adds the points again. This is exactly the Saturday bug.

Kafka's own documentation says that, unless you use its transactions, it guarantees at-least-once delivery by default. Many teams choose it too. A repeat can be caught and ignored, but a lost message is simply gone.

The diagram shows the same crash under both orders. Start at the question box and follow each answer to its result.

Exactly-once delivery is impossible over a real network because a sender cannot tell whether its message was lost or only the reply was lost. To be safe, it must send again, and that can create a duplicate.

Picture the payments service sending a purchase event to Kafka. It waits for the ack. Nothing comes back within its timeout, the time it is willing to wait. Two very different things could have happened.

• The event never reached Kafka. Sending again is the only way to avoid losing it.
• Kafka stored the event, but the ack got lost on the way back. Sending again creates a copy.

From the sender's side, both cases look the same: silence. It needs one rule for silence.

Retry, and you get at-least-once. Give up, and you get at-most-once. No third rule exists that the sender can follow alone.

This is a form of the Two Generals problem, an old puzzle in computer science. Two sides talking over a channel that can drop messages can never be fully sure they agree. Adding more confirmations never closes the gap. So when a standard or product promises exactly once, something beyond the network must be handling the repeats, as the next sections show.

Exactly once semantics means the effect of each message lands exactly once in the final result, even if the message was delivered or processed more than once. Some engineers prefer the name effectively-once, because it says what really happens.

The word semantics points at what you can observe afterwards, such as a wallet balance. It says nothing about how many times the bytes crossed the wire. A 2018 Apache Flink post defines it the same way: each event affects the final results exactly once.

Think of a polling booth on election day. Each voter has a serial number on the voter list. The officer cannot stop a voter from walking out and joining the queue again. So the officer ticks the serial number when the vote is cast.

When that voter reaches the front a second time, the officer sees the tick and sends them away. The voter arrived twice, but the vote counted once. Repeats are allowed to arrive, but their effect is not.

Map it back. The voter is the message, and the serial number is the event ID. The ticked list is a table of handled IDs, and the vote is the points update.

Where the picture breaks: one officer sees one voter at a time. Your workers run side by side, so two copies of one event can reach two workers at the same moment. That changes how the check must be built, as the next section shows.

Exactly-once systems you will meet rest on two parts:

• At-least-once delivery, so nothing is lost. Senders keep retrying until they get an ack.
• Either deduplication or an atomic commit at the receiver, so repeats change nothing. Deduplication spots a message you already handled and skips it. An atomic commit saves the output and the read position together, all or nothing, so a failed attempt leaves no visible result.

A change is idempotent when doing it twice leaves the same result as doing it once. Setting a balance to 500 points is idempotent. Adding 50 points is not, which is why the points worker needs a duplicate check.

You give every event a unique ID and store that ID in the same database transaction as the change the event causes. If the ID is already stored, the event is a repeat, and you skip it.

A transaction is a group of database writes that succeed together or fail together. That all-or-nothing rule is what closes the crash window behind the Saturday bug.

Go back to the polling booth. A tick made before the vote, followed by the voter fainting, means a lost vote. A vote with the tick made later lets a quick voter slip back first. Tick and vote must happen together.

Here is the fix the team ships. The payments service now puts an event_id, a unique ID created when the bill is paid, inside every purchase event. A new table, processed_events, makes event_id its primary key, so the database refuses a second copy.

The code below is complete and runs as it is, with Python's built-in sqlite3 module. It builds a small in-memory ledger, then delivers the same event twice, as a restart would.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE wallets (shopper_id TEXT PRIMARY KEY, points INTEGER)")
db.execute("CREATE TABLE processed_events (event_id TEXT PRIMARY KEY)")
db.execute("INSERT INTO wallets VALUES ('S1', 0)")
db.commit()

def handle(event):
    try:
        with db:  # commits both writes together, or rolls both back
            db.execute("INSERT INTO processed_events (event_id) VALUES (?)",
                       (event["event_id"],))
            db.execute("UPDATE wallets SET points = points + ? WHERE shopper_id = ?",
                       (event["points"], event["shopper_id"]))
    except sqlite3.IntegrityError:
        pass  # this event_id is already stored: a duplicate, so skip it
    # the real worker commits its Kafka offset here, after the ledger is safe

event = {"event_id": "bill-1001", "shopper_id": "S1", "points": 50}
handle(event)
handle(event)  # the same event again, as after a crash and restart
print(db.execute("SELECT points FROM wallets").fetchone()[0])

Run it and it prints 50, not 100. We ran it on Python 3.13.16. The second delivery found the ID already stored, so its points never landed.

What just happened? The first write in the with db block starts a transaction. If both writes succeed, the block commits them together. On the repeat, the INSERT hits the primary key and raises IntegrityError, so the block rolls back and the UPDATE never runs.

A real worker should also check that the UPDATE changed one row. An unknown shopper ID matches nothing and raises no error, so the event would count as handled with no points added.

In the real worker, the Kafka offset is committed after this function returns. A crash before that point means Kafka sends the event again. The stored ID quietly blocks the repeat. At-least-once delivery now produces exactly-once points.

The next diagram shows that path. Notice that both answers end with an offset commit, but only one of them touches the wallet.

We tested that gap with two threads handling the same event at once. With a separate lookup, the wallet ended at 100 points. With the unique key, it ended at 50, the right answer.

How long should you remember event IDs?

You cannot keep every ID forever, so a common choice is to remember IDs for a window of time. Amazon SQS FIFO queues, for example, promise that a retried send adds no duplicate if it comes within the 5-minute deduplication interval.

Your window must outlast your slowest possible repeat, with a safety margin added. That includes producer retries, consumer restarts and late uploads.

The team keeps purchase IDs for 30 days, because a shop's billing machine can go offline and upload a day of bills late. A nightly job deletes older rows, so the table stays small.

Kafka gives exactly-once inside Kafka through two features: an idempotent producer that drops repeated writes, and transactions that save outputs and consumer offsets together. Both arrived in Kafka 0.11.0.0.

The idempotent producer

With the idempotent producer, the broker gives each producer an ID, and the producer numbers every message it sends to a partition. If a retry arrives with a number the broker has already stored, the broker treats it as a duplicate.

That solves the lost-ack problem for writes into Kafka. There is a catch. A new producer instance gets a new ID, so on its own this check only works within one producer session.

Defaults depend on the client library. Apache Kafka's own Java client turned on idempotence and acks=all in 3.0. The setting acks=all means the broker confirms only after all in-sync replicas, the up-to-date copies of a partition, have the message.

A bug kept the idempotence default from applying in 3.0.0 and 3.1.0. It works from 3.0.1, 3.1.1 and 3.2.0. The Java client also turns idempotence off, logging only an INFO line, if you set a conflicting option, such as acks=1, without asking for idempotence.

The confluent-kafka library for Python is built on librdkafka, a C client. As of 19 October 2026, in version 2.16.0, its enable.idempotence still defaults to false. Set it to true yourself.

Transactions and fencing

Transactions handle the read, process and write loop. A worker reads from one topic, writes results to another, and commits its consumer offsets inside the same transaction. Either all of it becomes visible, or none does.

This works because Kafka stores consumer offsets as messages in an internal topic. So an offset commit can join a transaction like any other write.

Each transactional producer also has a fixed transactional.id. When a new copy starts with that ID, Kafka raises an epoch number, a generation counter, and fences off, or blocks, the older copy. A paused zombie worker that wakes up later cannot finish its stale transaction.

Readers must use the read_committed isolation level. An isolation level is a reader setting that decides whether it sees unfinished or cancelled writes.

With read_committed, readers skip messages from aborted transactions. The Java consumer's documented default is read_uncommitted, which shows everything, including writes that were later rolled back. As of 19 October 2026, librdkafka 2.16.0 defaults to read_committed.

The team later adds a second worker that turns purchases into points-earned events for other teams. Here is its core, using confluent-kafka. The helper to_points_event is a stand-in that builds the output message.

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "transactional.id": "points-events-worker-1",  # same ID after every restart
    "enable.idempotence": True,
})
producer.init_transactions()  # fences off any older copy with this ID

def process_batch(consumer, messages):
    producer.begin_transaction()
    for msg in messages:
        producer.produce("points-earned", value=to_points_event(msg))
    producer.send_offsets_to_transaction(
        consumer.position(consumer.assignment()),
        consumer.consumer_group_metadata())
    producer.commit_transaction()  # outputs and offsets become visible together

The init_transactions call is where fencing happens. The send_offsets_to_transaction call puts the consumer's progress inside the transaction. Then commit_transaction makes outputs and offsets visible in one step.

Kafka's documentation adds two consumer settings: enable.auto.commit set to false, and isolation.level set to read_committed. Without the first, the consumer could save progress outside the transaction.

If any step fails, call abort_transaction and move the consumer back to its last committed offsets before retrying. The consumer does not rewind on its own. In our test with confluent-kafka 2.16.0, on the test cluster built into librdkafka, its position stayed put after an abort. Kafka's documentation says the same.

By default, transactions need a cluster of at least three brokers. For a one-broker laptop setup, the sample server.properties in Kafka 4.3.1 already lowers transaction.state.log.replication.factor and transaction.state.log.min.isr to 1.

Kafka Streams, Kafka's own library for stream processing apps, wraps this whole pattern behind one setting. Set processing.guarantee to exactly_once_v2; its default in Kafka 4.3.1 is at_least_once.

To check it worked, read the output topic twice with the console consumer that ships with Kafka. Run kafka-console-consumer.sh with --bootstrap-server set to your broker, plus --topic points-earned, --from-beginning and --isolation-level read_committed. Only committed events should appear.

Run it again without the isolation flag, and its default, read_uncommitted, also prints events from aborted or still-open transactions. Any extra events in the second run are ones read_committed correctly hid.

Here is that worker as a picture. Look at the two arrows leaving the worker for the output topic and the offsets: they commit or roll back as one.

Where Kafka's guarantee stops

Kafka's exactly-once covers data that stays inside Kafka. Its documentation says exactly-once delivery to other systems generally needs their cooperation. It suggests storing the consumer's offset in the same place as its output.

The alternative is two-phase commit, a protocol where a coordinator asks every participant to prepare and then tells all of them to commit. Many output systems cannot take part in one. The points ledger follows the simpler advice: the event ID sits right next to the points it produced.

Stream processing engines use the same idea. The 2018 Flink post describes a two-phase commit sink base class added in Flink 1.4.0. Flink's Kafka sink uses it to pre-commit a Kafka transaction at each checkpoint, a snapshot of state and input position. It commits once the checkpoint completes.

Only if the outside service can spot repeats itself, usually through an idempotency key you send with each request. Otherwise you choose between a rare duplicate and a rare miss.

The loyalty app texts shoppers after every credit. The SMS gateway, the outside service that sends texts, cannot take a text back. The team decides a rare double text is fine, but a double cashback payout is not.

So each payout request carries the payout's own ID as an idempotency key, a unique value the partner uses to recognise a repeat. The bank partner treats a second request with the same key as the same payout. Here is the rule the team writes down:

• Database writes you own: deduplicate with a unique key in the same transaction.
• Partners that accept idempotency keys: send a stable key built from the event ID, never a fresh random one.
• Calls with no key, like most SMS sends: accept a rare repeat, or record each attempt and check its status before retrying.
• Money leaving the company: never retry blindly. Ask the partner for the status first, every time.

Choose by asking which hurts more for this data, a loss or a repeat, and how much extra work you can afford. Most products use different guarantees for different flows.

Kafka transactions also cost some waiting. A read_committed reader sees nothing past an open transaction in a partition, even later messages that were never part of one. So the worker must commit often enough that other teams are not left waiting.

Explain it as at-least-once delivery plus deduplication, and show exactly where the duplicate check lives.

1. 1

   Name the three guarantees

   Define at-most-once, at-least-once and exactly-once in one line each, with what each one can lose or repeat.

2. 2

   Explain why messages repeat

   Use the lost ack. A sender cannot tell a lost message from a lost reply, so it retries.

3. 3

   Choose at-least-once delivery

   Nothing may be lost, so producers retry and consumers commit offsets only after the work is done.

4. 4

   Make the effect idempotent

   Add a unique event ID and store it in the same transaction as the change. Point at the unique key.

5. 5

   Draw the boundaries

   Say where a tool's exactly-once ends, such as Kafka transactions, and how each outside call is protected.

6. 6

   Size the window

   Keep IDs longer than the slowest possible repeat, and say how old IDs get cleaned up.