How Kafka Works Internally: Log, Partitions and Replicas

When many programs share one stream of events through a plain queue, they steal events from each other and lose the history. A queue hands each event to one reader and then deletes it. Kafka internals are built to avoid exactly that.

Picture a common setup. A city bus operator in Nagpur fits a small box in each of its 1,200 buses. Each box sends a GPS position every 5 seconds, plus an event whenever the driver brakes hard. The setup is made up, but the problems are real.

That is about 240 GPS events a second. The team first ran everything through one server with a simple queue. Within a month, three things went wrong.

• The live map and the weekly safety report read from the same queue. Each event went to only one of them, so the report quietly missed about half the hard braking.
• The safety team wanted to rerun last week's report with a new rule. The events were already gone, because the queue deletes an event once someone reads it.
• The queue server's disk failed one night, and about 40 minutes of braking events (made up) were lost for good.

Kafka fixes all three, and this guide shows how, one part at a time. The same bus pipeline runs through every section. For when to pick Kafka over a classic queue or a cloud queue service, our message queues guide compares them.

You need six words: broker, topic, partition, offset, replica and consumer group. Each gets a plain meaning and an example from the bus pipeline.

Broker. A broker is one Kafka server that stores data and answers requests. A cluster is a group of brokers working together. Example: the bus team runs three brokers, numbered 1, 2 and 3.

Topic. A topic is a named stream of records, like a folder for one kind of event. A record is one event with an optional key, a value and a timestamp. Example: the topic bus-events holds every GPS ping and braking event.

Partition. A partition is one slice of a topic, stored as its own log on a broker. Example: bus-events has 6 partitions, numbered 0 to 5, spread across the three brokers.

Offset. An offset is the position number of a record inside one partition. Each new record gets the next number up. Example: the record at offset 81,402 in partition 3 is a GPS ping from bus MH31-4417.

Replica. A replica is one copy of a partition, and Kafka keeps each copy on a different broker. Example: partition 3 has three replicas, one on each broker, so one dead disk loses nothing.

Consumer group. A consumer group is a team of reader programs that share the work of reading a topic. Example: the live map is one group and the safety report is another. A producer is any program that writes records.

Kafka works internally as a set of append-only logs. Producers add records to the end, brokers keep them for a set time, and consumers read forward from a position they choose. Nothing is removed just because someone read it.

An append-only log is a list you can only add to at the end. You never edit or remove a record in the middle. The original 2011 Kafka paper from LinkedIn describes each partition as a logical log, and the idea still holds.

Picture a bead-mala stringer at a stall in Itwari market. Beads arrive one by one, and she threads each onto the end of a string, in order. Each string is a partition, and each bead is a record.

Customers who count the beads do not take them away. Each customer remembers the last bead they counted and carries on from the next one. That is how consumers use offsets, and why many readers can share one log.

Where the picture breaks: a real string never runs out, but Kafka deletes old beads from the far end after a set time. A customer who comes back too late finds their place gone.

Here is the bus pipeline on Kafka. Start at the buses and follow the arrows into the topic, then to three readers. Each reader is its own consumer group, so each one sees every event.

Kafka consumers pull data from the broker; the broker does not push it. The Kafka design documentation explains that pulling lets a consumer take large batches and catch up at its own pace.

That fixes two of the three problems. The live map and the safety report no longer steal events from each other. Because events stay for a set time, the safety team can rerun last week by starting from an older offset.

When a record has a key, the producer hashes the key and uses the result to pick a partition, so the same key lands in the same partition. A hash is a function that turns any input into a number in a repeatable way.

The bus team uses the bus number as the key, so every event from bus MH31-4417 lands in the same partition. That matters because Kafka keeps order only inside one partition. Across partitions, there is no promised order.

For a bus, order is everything. If a GPS ping from 9:02 is read after one from 9:03, the bus jumps backwards on the live map. With one key per bus, each bus's events stay in the order they were written.

The small Python sketch below shows the idea with an in-memory list per partition. It uses crc32, a hash from Python's standard library. Later we show which real clients use which hash.

import zlib

NUM_PARTITIONS = 6
log = {p: [] for p in range(NUM_PARTITIONS)}   # one list per partition

def pick_partition(key):
    return zlib.crc32(key.encode()) % NUM_PARTITIONS

def append(key, value):
    p = pick_partition(key)
    log[p].append((key, value))
    offset = len(log[p]) - 1                     # next number in this partition
    return p, offset

print(append("MH31-4417", "gps 21.1458,79.0882"))
print(append("MH31-4417", "hard_brake 0.62g"))

What just happened? The modulo (the % sign, which gives the remainder) maps any key to one of the 6 partitions. The offset line shows that offsets count within one partition. Running it prints (5, 0) then (5, 1): same partition, offsets 0 and 1.

Kafka stores each partition as a folder of files on disk, called segments, and always writes to the end of the newest one. Old segments are deleted once they pass a time or size limit, or cleaned by compaction.

The 2011 paper describes a log as segment files of roughly equal size, with 1GB as its example. In Kafka 4.3.1 the default segment size, log.segment.bytes, is 1 GiB (1,073,741,824 bytes). Kafka starts a new segment when the current one is full, or after segment.ms, 7 days by default.

Writing only at the end sounds simple, and that is the point. The design documentation explains that disks are far faster at reading and writing in a straight line than at jumping around.

Kafka also leans on the page cache. The page cache is memory the operating system uses to keep recently used file data. Fresh records are often still in memory when consumers ask for them, so many reads never touch the disk.

Retention: how long records stay

Retention is the rule for when Kafka deletes old data, set by time or by size. Kafka removes whole old segments, not single records. The broker default, log.retention.hours, is 168 hours (7 days). The bus team sets bus-events to 14 days, enough to rerun two weekly reports.

Compaction: keeping the latest value per key

Log compaction is a cleanup mode that keeps at least the last value for each key and drops older ones. The bus team adds a second topic, bus-latest-position, keyed by bus number. After compaction, it still holds the newest position of every bus.

A new live map server can read that small topic from the start and learn where all 1,200 buses are. When a bus is retired, the team writes a record with its key and an empty value, called a tombstone.

Compaction then removes older records for that key. The tombstone itself is cleaned out later, after delete.retention.ms, which defaults to 24 hours. A reader that is more than a day behind can miss the delete.

Kafka replication works by keeping copies of each partition on several brokers: one leader takes all writes, and followers fetch new records from it. A record is committed once every replica in the in-sync set has it.

Go back to the bead stringer. Apprentices at other tables each keep a copy of one of her strings, adding the same beads in the same order. If her table is knocked over, an apprentice whose copy is complete takes over.

Where the picture breaks: apprentices are not handed beads. Kafka followers ask the leader for new records, just as a normal consumer would, and receive them in the order the leader wrote them.

In-sync replicas and the high watermark

The in-sync replica set, or ISR, is the list of replicas that are caught up with the leader. A follower that has not caught up for too long is removed. The setting replica.lag.time.max.ms sets how long; its default is 30,000 ms (30 seconds).

The high watermark is the offset up to which every ISR member has the data. Records below it are committed. Consumers are only given committed records, so a reader never sees a record that a normal failover could later erase.

Here is what happens when the ingest service writes one braking event to partition 3. Start at the producer. The followers' arrows point at the leader, because followers pull.

acks and min.insync.replicas

The producer setting acks says how long the producer waits before it treats a write as done. The table compares the three choices on the bus cluster, with 3 replicas per partition.

In Kafka 4.3.1, the Java producer defaults to acks=all with idempotence on, which stops its own retries from writing duplicates. librdkafka also defaults to acks=all, but its enable.idempotence defaults to false, as of version 2.16.0.

There is a catch with acks=all. It waits for the current ISR, and the ISR can shrink. If both followers fall out, the leader alone is the ISR, and acks=all behaves like acks=1.

The topic setting min.insync.replicas closes that gap, and its default is 1. The bus team sets it to 2. Now the leader rejects acks=all writes with a NotEnoughReplicas error when fewer than 2 replicas are in sync.

Replication keeps a record safe once it is in Kafka. It does not decide how often your code processes it. Retries can still create duplicates, which our delivery guarantees guide covers with idempotent producers and transactions.

When a broker fails, the Kafka controller notices and makes a safe follower the new leader for each partition that broker was leading. Producers and consumers then find the new leader and carry on.

The controller is the part of Kafka that tracks which broker leads which partition. Kafka 4.0 was the first major release to run entirely without Apache ZooKeeper, a separate coordination service that older versions relied on. The controller now runs in KRaft mode.

KRaft is Kafka's own version of Raft, a consensus method that lets a few servers agree on one ordered log. A small group of controller nodes keeps the cluster's metadata in that log. Metadata means facts about the cluster, such as topics, partitions and each ISR.

The KIP-500 proposal explains the motivation: with ZooKeeper, administrators had to learn, deploy and manage two separate distributed systems. Today each broker sends a heartbeat to the active controller every 2 seconds by default. After 9 seconds with none, the controller fences the broker, meaning it stops treating it as alive.

Follow the diagram from the top. Broker 1 has died, and it was the leader for partition 3. The question box is the key decision.

In a normal election, only a replica known to hold every committed record can become leader. That is an in-sync replica, or on newer clusters one from a backup list explained in the engineer section. That is why the high watermark rule works. Nothing a consumer has seen disappears in a normal failover.

Pause and predict: every in-sync replica is gone, and one out-of-sync replica is alive. Should Kafka promote it to bring the partition back?

By default it does not. Since version 0.11.0.0, Kafka waits for a consistent replica. The setting unclean.leader.election.enable, false by default, lets an out-of-sync replica take over. The partition returns sooner, but committed records it never copied are lost.

A consumer group shares a topic's partitions among its members, giving each partition to exactly one member at a time. Each group saves its own offsets, so different groups read the same data independently.

In the bead picture, each customer is a group. Two customers can count the same string, each keeping their own place. But when one family shares the work, the strings are split up, so no string is counted twice in that family.

Look at the diagram. The alerts group has two members sharing six partitions. The safety report group has one member that reads all six on its own.

Committing an offset means saving the position a group has finished up to. Kafka stores each committed position as a record in an internal topic named __consumer_offsets. So Kafka keeps its own bookmarks in a Kafka log.

Tracking one number per partition is cheap, and it makes replay easy. The safety report can move its offset back to last Monday and read again. That fixes the second problem from the opening.

Here is the alerts reader in Python, using the confluent-kafka client library. send_alert is a stand-in for the code that messages depot staff.

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "kafka1:9092,kafka2:9092,kafka3:9092",
    "group.id": "alerts",
    "enable.auto.commit": False,       # we commit by hand
    "auto.offset.reset": "earliest",   # first run: start at the oldest record
})
consumer.subscribe(["bus-events"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    if msg.value() and msg.value().startswith(b"hard_brake"):
        send_alert(msg.key(), msg.value())
    consumer.commit(message=msg, asynchronous=False)

What just happened? The group.id line puts this reader in the alerts group. Turning off auto commit and committing after send_alert saves the offset only after the work is done. The committed value is the next offset to read.

We ran this loop on librdkafka's built-in test cluster and killed it right after one alert, before the commit. On restart it read that braking event again and sent the alert twice. With this pattern, a crash causes a repeat, not a skip.

Rebalancing and the partition limit

A rebalance is when Kafka reshuffles partitions among a group's members because one joined, left or crashed. Kafka 4.0 made the new rebalance protocol from KIP-848 generally available. Its release announcement says goodbye to stop-the-world rebalances. In the old protocol, every rebalance needed the whole group to rejoin.

Consumers must opt in with group.protocol=consumer. As of Kafka 4.3.1 and librdkafka 2.16.0, the client default is still classic, the old protocol. The librdkafka 2.16.0 docs say this default will change to consumer in later releases.

Because each partition has only one reader per group, the partition count caps how many members do useful work. If the alerts group runs 8 readers on 6 partitions, 2 sit idle.

Share groups, from KIP-932, lift that cap for queue-style work: several readers share one partition and acknowledge records one by one. They were a preview in Kafka 4.1 and are production-ready from 4.2.

You check them with two command-line tools that ship with Kafka: kafka-topics.sh shows leaders and the ISR, and kafka-consumer-groups.sh shows offsets and lag. Run them from the Kafka folder on any machine that can reach a broker.

# Leader, replicas and in-sync replicas for every partition
bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --topic bus-events

# Only partitions whose ISR is below min.insync.replicas, or with no leader (empty output is good)
bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-min-isr-partitions

# Committed offset, newest offset and lag for each partition of one group
bin/kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group alerts

In Kafka 4.3.1, the first command prints one line per partition with tab-separated fields. On a healthy bus cluster, partition 3 might show Leader: 1, Replicas: 1,2,3 and Isr: 1,2,3.

Two more fields follow, Elr and LastKnownElr, which are explained in the next section. If Isr lists fewer brokers than Replicas, a follower has fallen behind or its broker is down.

The group command prints one row per partition with CURRENT-OFFSET, LOG-END-OFFSET and LAG columns. LAG is LOG-END-OFFSET minus CURRENT-OFFSET, counted in offsets, which is roughly records, not bytes or seconds. In this tool, LOG-END-OFFSET is the high watermark, not the leader's newest write.

They make min.insync.replicas matter beyond acks=all. The high watermark stops moving while the ISR is smaller than min.insync.replicas. And in new clusters since 4.1, the controller tracks eligible leader replicas, ELR for short.

The strict min ISR rule

In the Kafka 4.3.1 source, the leader skips moving the high watermark while the partition is under min ISR. The documentation calls this the strict min ISR rule. It is not new in 4.x: the same check is in the 3.9.0 source.

This changes what acks=1 means. With min.insync.replicas of 2 and only the leader in sync, an acks=1 write still succeeds. But consumers cannot read it until a follower catches up and the high watermark moves.

So a producer can see success while every consumer sees nothing new. If the bus team's GPS producer uses acks=1, the live map can freeze while both followers are out of the ISR, with no producer error at all. Lag will not warn you either: the high watermark is stuck, so LOG-END-OFFSET stops and lag drops to 0. The --under-min-isr-partitions command shows the problem.

Eligible leader replicas

Eligible leader replicas are replicas that left the ISR once it was smaller than min.insync.replicas. Since the high watermark could not move then, they still hold every committed record. The controller keeps them in a list called the ELR.

The documented election order is: a live ISR member first, then an unfenced ELR member, then the last known leader once it is back. In the source, a replica that shut down uncleanly is left out of the ELR.

ELR is on by default only in clusters created on Kafka 4.1 or later. An upgraded cluster turns it on with kafka-features.sh, using upgrade --feature eligible.leader.replicas.version=1. Enabling it moves min.insync.replicas to cluster level: broker-level values are removed and can no longer be changed. Changing min.insync.replicas clears the current ELR state.

They use different hash functions by default. The Java producer hashes keys with murmur2. librdkafka, the C library inside the confluent-kafka Python client, defaults to a partitioner called consistent_random that uses CRC32.

We ran a real librdkafka 2.16.0 producer against its built-in test cluster with 8 bus ids. With the default partitioner, all 8 landed where crc32 modulo the partition count predicts, and none matched murmur2.

We also computed both hashes for 1,200 made-up bus ids on 6 partitions. The two defaults disagreed for 1,004 ids (83.7%). Picture an old Java ingest service and a new confluent-kafka Python one writing the same bus. That bus's events split across two partitions and lose their order.

The fix is one setting. Set partitioner to murmur2_random in librdkafka, which its documentation calls Java Producer compatible. In our test, all 8 keys then matched murmur2, with 0 matching crc32.

The common mistakes come from expecting Kafka to behave like a database or a simple queue.

• Expecting order across the whole topic. Kafka orders records only inside one partition, so pick a key that groups the records that must stay in order.
• Using one replica to save disk. With a replication factor of 1, a single dead disk loses the partition for good.
• Setting acks=all and leaving min.insync.replicas at its default of 1. When followers fall out of the ISR, the leader alone can confirm writes.
• Committing offsets before the work is done. A crash then skips records that were never handled.
• Choosing too few partitions. The partition count caps how many readers in one group can work at once.
• Mixing Java and librdkafka producers on one keyed topic without setting the same partitioner in both.

Explain Kafka as a log first, then add partitions, replication and consumer groups, and tie each one to a requirement in the question.

1. 1

   Start with the log

   Say that a topic is an append-only log, records get offsets, and reading does not delete. Mention retention by time or size.

2. 2

   Pick the key

   Name the key and why. For bus events, the bus number keeps each bus's events in order, because order holds only inside a partition.

3. 3

   Size the partitions

   Estimate the peak write rate and the most readers any group needs. Choose a partition count that covers both, with room to grow.

4. 4

   Set durability

   Say three replicas, min.insync.replicas of 2 and acks=all for data that must not be lost. Explain the ISR and the high watermark in one sentence each.

5. 5

   Walk through a failure

   Kill a broker out loud. The controller promotes an in-sync follower, clients refresh metadata, and committed data survives.

6. 6

   Add the readers

   Give each use its own consumer group, commit offsets after processing, and watch consumer lag.