Message Queues from First Principles: Kafka vs RabbitMQ vs SQS

Say you are building order placement for a food delivery app. A customer taps "Place order" and five things need to happen: save the order, capture the payment, ping the restaurant, find a rider and send a confirmation email. The first version every team writes is one HTTP handler that calls all of them, one after another.

It works on your laptop. Then Friday dinner rush hits, rider matching slows to a crawl, and checkout is slow for everyone, even though the order was saved in 40 milliseconds. This article builds message queues up from that problem, then compares Kafka, RabbitMQ and SQS honestly so you can pick one in a design review or an interview.

Synchronous fan-out breaks because latency adds up, availability multiplies down, and one slow dependency holds the whole request hostage. Here is the naive handler with made-up but realistic timings.

POST /orders  (synchronous version, example timings)
  saveOrder()            40 ms
  payment.capture()     300 ms
  restaurant.notify()   200 ms
  rider.assign()        900 ms   <- searching nearby riders
  email.sendReceipt()   400 ms
  ---------------------------------
  total               ~1840 ms, and it fails if ANY call fails

Three problems hide in that list. First, latency: the customer waits for the slowest chain of calls, including work they do not care about right now, like the email. Second, availability: if four downstream services each have 99.9% uptime, the chain is up only about 0.999 to the power of 4, roughly 99.6% of the time. That is around 35 hours of failed checkouts a year instead of under 9.

Third, partial failure. Payment was captured, then the restaurant call timed out. Retry everything and risk charging twice? Refund? Nobody writes that compensation logic correctly at 2 a.m.

A message queue is a durable buffer between services: a producer writes a message, the broker stores it, and a consumer reads and processes it later, at its own pace. The producer only needs the broker to say "stored". It does not need the consumer to be up, fast or even deployed yet.

Rewrite the order flow with that idea. The API saves the order and publishes one event, "order placed", then returns to the customer. Each downstream service subscribes and does its part independently.

That one change buys you three things:

• Decoupling: if the email worker is down for ten minutes, events wait and get processed when it returns. Checkout never noticed.
• Buffering spikes: an IPL final or a flash sale sends 20x normal traffic. The queue absorbs the burst and consumers drain it at a rate the database can handle.
• Retries: a failed message is not lost. The broker redelivers it, so a transient 503 from an SMS provider heals itself.

The cost is real too. The system is now eventually consistent: for a moment the order exists but no rider is assigned. You have new infrastructure to run, and debugging means following a message across services instead of reading one stack trace.

A queue gives each message to one consumer, pub/sub gives each subscriber its own copy, and a log keeps every message so any consumer can read or re-read it by position. People say "queue" for all three, which is where most confusion in Kafka vs RabbitMQ debates comes from.

• Point-to-point queue: messages are work items. Ten workers share one queue, each message goes to one of them and is deleted once acknowledged. Think "generate this invoice PDF".
• Publish/subscribe: one event, many interested parties. Restaurant, rider and email services each get their own copy of order.placed. In RabbitMQ that is a fanout or topic exchange bound to several queues; on AWS it is usually SNS fanning out to several SQS queues.
• Log (event stream): an append-only, ordered, retained sequence. Consumers delete nothing; each remembers an offset, its position in the log, so a new analytics service can replay last week. This is Kafka, and also RabbitMQ streams.

The practical test: after processing, should the message disappear? If yes, use a queue. If future services or a bug-fix replay might need it, use a log.

Delivery guarantees describe what happens when something crashes between receiving a message and recording that it was processed. The whole story is about when the consumer acknowledges.

• At-most-once: acknowledge first, then process. If the consumer crashes mid-way, the message is gone. Messages may be lost but are never redelivered. Fine for metrics pings, terrible for payments.
• At-least-once: process first, then acknowledge. If the consumer crashes after processing but before the ack, the broker redelivers. Nothing is lost, but duplicates happen. This is the default for SQS standard queues, RabbitMQ with manual acks and Kafka consumers that commit offsets after processing.
• Exactly-once: each message affects the result once. This is what everyone wants and what no network can promise by itself.

Kafka does offer exactly-once semantics, and it is worth knowing what that means precisely. An idempotent producer gets a producer ID and sequence numbers so the broker drops duplicate writes caused by retries. Transactions let a producer write to several partitions and commit consumer offsets atomically, so a read-process-write pipeline inside Kafka either fully happens or does not.

The boundary is Kafka itself. The moment your consumer calls a payment gateway or sends an SMS, Kafka cannot roll that back. Kafka's own design docs suggest storing the consumer's offset in the same place as its output, so both are committed together. SQS FIFO is similar: it deduplicates sends that repeat within a 5-minute deduplication interval, but if your consumer takes longer than the visibility timeout, another consumer can still receive the same message.

An idempotent consumer is one where processing the same message twice leaves the system in the same state as processing it once. The most dependable way is a dedupe table written in the same database transaction as the business change.

CREATE TABLE processed_messages (
  consumer     text        NOT NULL,
  message_key  text        NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, message_key)
);

import { Pool } from 'pg';

const pool = new Pool();

type OrderPlaced = { eventId: string; orderId: string; restaurantId: string };

export async function handleOrderPlaced(msg: OrderPlaced) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // 1. Claim the message. A duplicate hits the primary key and inserts nothing.
    const claim = await client.query(
      `INSERT INTO processed_messages (consumer, message_key)
       VALUES ('restaurant-notifier', $1)
       ON CONFLICT DO NOTHING`,
      [msg.eventId],
    );
    if (claim.rowCount === 0) {
      await client.query('ROLLBACK');
      return; // already handled: just ack it
    }

    // 2. Business change in the SAME transaction.
    await client.query(
      `INSERT INTO kitchen_tickets (order_id, restaurant_id, status)
       VALUES ($1, $2, 'NEW')`,
      [msg.orderId, msg.restaurantId],
    );

    await client.query('COMMIT'); // both rows or neither
  } catch (err) {
    await client.query('ROLLBACK');
    throw err; // no ack, so the broker will redeliver
  } finally {
    client.release();
  }
}

Because the claim row and the kitchen ticket commit together, a crash anywhere leaves either both or neither. Ack the message only after COMMIT returns. Keep dedupe rows for longer than your longest possible redelivery window, then clean them up with a scheduled job.

Pick the key carefully. A broker message ID changes if the producer re-sends the event as a new message, so prefer a producer-generated event ID or a business key like orderId plus event type. For side effects outside your database, pass the key along: most payment APIs accept an idempotency key, so a retried capture returns the original result instead of charging again.

Brokers guarantee order only within a unit of parallelism: a Kafka partition, an SQS FIFO message group or a single RabbitMQ queue with one consumer. Global ordering across everything would force one consumer to process messages one at a time, which kills throughput.

The trick is that you rarely need global order. You need order per entity: "order placed" must be handled before "order cancelled" for the same order, but two different orders can be processed in any order. So you choose a key. In Kafka, the producer hashes the message key (say, orderId) to pick a partition, and every event for that order lands in the same partition, in order. In SQS FIFO, the MessageGroupId plays the same role.

A consumer group is a set of consumers that share the work of reading a topic; Kafka assigns each partition to exactly one consumer in the group. Different groups read independently, each tracking its own offsets, which is how one topic serves both the rider service and the email service.

Two consequences trip people up. Your maximum parallelism within a group is the partition count: with 3 partitions, a fourth rider consumer sits idle. And a bad key choice creates a hot partition. Key by restaurantId during a big chain's promo and one partition takes most of the traffic while the others idle.

Kafka 4.2 (February 2026) made share groups, also called Queues for Kafka, production-ready: many consumers share the same partitions with per-message acks, like a classic queue, but without per-key ordering. For keyed, ordered events, stick with consumer groups.

Backpressure is how a system slows producers or limits intake when consumers fall behind, instead of letting work pile up until something falls over. A queue gives you a buffer, not infinite capacity, so you need to know how big the buffer is and what happens when it fills.

• Watch the right metric: queue depth and age of the oldest message for SQS and RabbitMQ, consumer lag for Kafka. Alert on age, not just count; 10,000 fast messages is fine, 50 messages that are 20 minutes old is not.
• Limit in-flight work: RabbitMQ consumers set a prefetch count so the broker never pushes more unacked messages than a worker can handle. Kafka and SQS consumers pull, so they take only what they ask for.
• Scale consumers on lag, but remember the database behind them has its own limit: ten times the consumers can mean ten times the connections.
• Know your runway: SQS keeps messages for 4 days by default (up to 14), and Kafka deletes by retention time or size. If consumers stay down longer than retention, data is gone.

Retries handle temporary failures; a dead-letter queue (DLQ) is a separate queue where messages go after they fail too many times, so they stop blocking or burning resources and a human can inspect them. You need both, because not every failure is temporary.

Split errors into two kinds. Transient errors (timeouts, 503s, a deadlock) deserve retries. Permanent errors (a malformed payload, a restaurant ID that does not exist, a bug in your parser) will fail every single time. A message that always fails is called a poison message, and without a cap it gets redelivered forever, eating consumer capacity and, on ordered queues, blocking everything behind it.

// Exponential backoff with full jitter: spread retries out so
// a thousand failed consumers do not hammer a recovering service in sync.
export function backoffMs(attempt: number, baseMs = 500, capMs = 60_000) {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * exp);
}

// attempt 0 -> 0-500 ms, 1 -> 0-1 s, 2 -> 0-2 s ... capped at 60 s

Each broker wires this up differently. In SQS, a received message becomes invisible for the visibility timeout (30 seconds by default); if you do not delete it in time, it reappears. A redrive policy with maxReceiveCount moves it to the DLQ after that many receives. In RabbitMQ, you reject the message and a dead-letter exchange routes it onward; quorum queues also enforce a delivery limit, which defaults to 20 since RabbitMQ 4.0. In Kafka, the broker has no DLQ for plain consumers, so teams publish failures to retry topics and a dead-letter topic themselves (Kafka Connect and libraries like Spring Kafka have this built in).

The outbox pattern writes the event into a table in the same database transaction as the business change, then a separate relay publishes it to the broker. It exists because of the dual-write problem: writing to a database and publishing to a broker are two separate systems, and you cannot commit both atomically.

Look at the order API again. If it inserts the order and then the publish fails, the order exists but no restaurant ever hears about it. Publish first and then the insert fails, and a rider shows up for an order that does not exist. Wrapping both in try/catch does not help; the process can die between the two lines.

BEGIN;

INSERT INTO orders (id, customer_id, restaurant_id, total_paise, status)
VALUES ('ord_81f2', 'cus_19', 'res_44', 48900, 'PLACED');

INSERT INTO outbox (id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'ord_81f2', 'order.placed',
        '{"orderId":"ord_81f2","restaurantId":"res_44"}');

COMMIT;  -- the order and its event exist together, or not at all

The relay comes in two flavours. A polling publisher selects unsent rows every few hundred milliseconds: simple, and fine at moderate volume. Transaction log tailing (change data capture, often with Debezium) reads the write-ahead log for lower latency, but is another system to operate. Either way, the relay can crash after publishing and before marking the row sent, so the event goes out twice. Idempotent consumers close the loop.

Use Kafka when events are a shared, replayable stream that many teams read at high volume; use RabbitMQ when you need flexible routing of tasks to workers with per-message acks; use SQS when you are on AWS and want a queue with no servers to run. Each one can be bent into the others' shape, but you pay for it.

When Kafka is the right call

Pick Kafka when the event itself is the product: clickstreams, order events that feed analytics, fraud and search indexing, change data capture from databases. Retention and replay mean a new team can plug in next month and backfill. Kafka 4.0 (March 2025) removed ZooKeeper entirely, so clusters now run in KRaft mode only, which makes self-hosting simpler than it used to be. It is still the most operationally demanding of the three, so a two-person startup sending emails does not need it.

When RabbitMQ is the right call

Pick RabbitMQ when you have work to distribute and interesting routing rules: route "payment.failed.upi" to one queue and "payment.*" to an audit queue, set per-queue TTLs, use priorities. For anything you cannot lose, use quorum queues, which replicate with Raft; classic queue mirroring was removed in RabbitMQ 4.0. If you also want replay, RabbitMQ streams give you a log inside the same broker.

When SQS (and SQS FIFO) is the right call

Pick SQS when you are on AWS and want decoupling without running anything. Standard queues scale to nearly unlimited throughput with at-least-once delivery and best-effort ordering. Messages can be up to 1 MiB. SQS FIFO adds per-group ordering and send deduplication, but watch the limits: without high throughput mode, each partition handles 300 transactions per second per API action, or 3,000 messages per second with batches of 10. High throughput mode goes much higher, though the ceiling depends on region; Asia Pacific (Mumbai) is listed at up to 9,000 transactions per second, against 70,000 in the largest US and EU regions.

• Assuming no duplicates. Every at-least-once system delivers them eventually, usually during a deploy or a rebalance.
• Acking before the work is done. Auto-ack or committing the Kafka offset before processing silently turns at-least-once into at-most-once.
• Visibility timeout shorter than processing time. The message reappears while the first worker is still busy, and now two workers process it in parallel.
• Publishing to the broker inside a database transaction. If the transaction rolls back, the event already went out. Use the outbox.
• Retrying permanent errors forever, with no backoff and no DLQ. One bad payload pins a consumer at 100% CPU.
• Using a queue where a synchronous call is right. If the user needs the answer now, like "was my UPI payment approved?", queuing it just adds latency and polling.

A strong food delivery answer sounds like this: payment authorisation stays synchronous because the user needs the result; everything after "order placed" is an event published through an outbox; consumers are idempotent on eventId; events are keyed by orderId so changes to one order stay ordered; failures retry with jittered backoff and land in an alarmed DLQ after five attempts. That pre-empts most follow-up questions.

If you remember one thing, make it this: a queue does not remove failure, it moves failure somewhere you can handle it calmly. Take the food delivery example, sketch it with an outbox, keyed events, idempotent consumers and a DLQ, and you will be able to defend every box on the board.