Raft Consensus Explained: Leader Election and Log Replication

When the network splits, a backup server cannot tell a dead primary from one it simply cannot reach. If it takes over anyway, two primaries accept different writes. Raft consensus exists to stop exactly that.

Picture a common setup. A cloud platform team keeps its config in one database: which rack holds each disk, which firewall rules each machine has, which IP address belongs to whom. A backup database sits in a second building. The setup is made up, but the problem is real.

A health-check script watches the primary. If the primary does not answer for 10 seconds, the script promotes the backup to primary.

One night, a fibre cut splits the two buildings. The primary is healthy, but the script can no longer reach it, so it promotes the backup. For 6 minutes, both databases accept writes.

• One customer opens port 443 on 40 machines, and that change lands on the old primary.
• Another customer closes port 22 on 120 machines, and that change lands on the new primary.
• When the fibre returns, the two databases disagree about 160 machines, and nobody can say which version is right.

This is split brain: two servers that each believe they are in charge. The naive fix is a longer timeout, say 60 seconds instead of 10. It only moves the problem, because a 61-second split does the same damage and a real crash now takes a minute to handle.

So the team sets one rule for the new config store: it must never have two bosses, even during a split. It picks Raft and runs the store on five servers, named s1 to s5, spread over three zones. A zone here is a separate building with its own power and network links.

State machine replication means running the same program on several servers and feeding every copy the same commands in the same order. If every command is deterministic, every copy ends in the same state.

A state machine is any program whose data, called its state, changes only when it runs a command. In the config store, the state machine is a simple key-value map. A command such as set firewall/vm-4512 to allow-443 changes one key.

A deterministic command gives the same result every time it runs on the same state. Setting a key is deterministic. Reading the clock or picking a random number is not, so the store works out those values once, before the command enters the log.

The replicated log is a numbered list of commands, kept on every server. Entry 1 comes before entry 2, which comes before entry 3. A consensus algorithm's whole job is to make every server's log hold the same entries in the same order.

Think of a housing society's minutes book. Every decision, such as raising the maintenance fee, goes in as a numbered item, and each committee member keeps a copy. Here is how the picture maps onto Raft:

• The minutes book is the replicated log.
• One numbered item is one log entry.
• The society's current rules are the state machine's data, rebuilt by reading from item 1.
• The secretary, the only person who writes new items, is the leader.

Where the picture breaks: a society can reopen an item and argue about it. In Raft, a committed entry never changes, and servers only agree on each entry's position, never on whether the command is wise.

Notice that only the leader adds entries, and entries flow only from the leader outward. That one-way flow is a big reason Raft is easier to follow.

Raft consensus is an algorithm that keeps a replicated log identical across a group of servers. It keeps accepting writes as long as a majority of them are running and can talk. Diego Ongaro and John Ousterhout at Stanford published it in 2014.

Consensus means agreement: all servers agree on which command sits at each position in the log. Paxos is an older consensus algorithm with a reputation for being hard to understand. The Raft paper notes that most consensus implementations of its time were based on Paxos or influenced by it.

Raft was designed to do the same job while being easier to learn. The paper backs this with a user study: of 43 students who learned both, 33 answered questions about Raft better than questions about Paxos.

Raft splits the problem into three parts you can learn one at a time:

• Leader election: pick at most one leader per term, and replace it quickly when it fails.
• Log replication: the leader takes client writes and copies them into the followers' logs.
• Safety: make sure a committed entry is never lost or changed, even across many leader changes.

Every server plays one of three roles. A follower is passive and answers the leader. A candidate is asking the others for votes. A leader takes all client writes and sends heartbeats.

Raft counts time in terms. A term is a numbered period that starts with an election, like term 7 or term 8, and has at most one leader. Every message carries the sender's term.

A server rejects any request that carries an older term than its own. A leader or candidate that sees a newer term steps back to follower at once.

In Raft leader election, a follower that hears no heartbeat for a random timeout becomes a candidate. It raises the term by one, votes for itself and asks the others for votes. Votes from a majority make it leader.

The election timeout is how long a follower waits without hearing from a leader. The Raft paper recommends a conservative timeout such as 150 to 300 milliseconds, picked at random from that range. Usually one server times out first, wins, and sends heartbeats before anyone else wakes up.

Without that randomness, servers time out together and split the votes, so nobody wins. In the paper's tests, elections with no randomness consistently took longer than 10 seconds. Adding just 5 milliseconds of randomness cut the median downtime to 287 milliseconds.

Real systems tune these numbers to their network. etcd's tuning guide gives defaults of a 100 millisecond heartbeat and a 1,000 millisecond election timeout, as of its v3.7 docs on 19 October 2026.

Two voting rules keep elections safe. Each server gives at most one vote per term, first come, first served. A server also refuses any candidate whose log is less up to date than its own.

This diagram shows the three roles and what moves a server between them. Start at Follower and follow the Yes arrows to Leader.

Here is one failover in the config store. The timings are made up but realistic.

1. 1

   Term 7 runs normally

   s1 is leader and sends heartbeats every 50 milliseconds. s2 to s5 reset their election timers each time one arrives.

2. 2

   s1's disk controller dies

   s1 stops sending anything, and the followers' timers run down. s3 drew the shortest random timeout, 180 milliseconds, so it runs out first.

3. 3

   s3 becomes a candidate

   It raises its term to 8, votes for itself, saves both to disk, and sends a vote request to s1, s2, s4 and s5.

4. 4

   Votes come back

   s2 and s4 have not voted in term 8, and s3's log is at least as up to date as theirs. Both say yes, so s3 holds 3 of 5 votes, counting its own.

5. 5

   s3 takes charge

   s3 is leader for term 8. It sends heartbeats at once, so s5 resets its timer before it can start a rival election.

The vote rule in Python

Here is how a server answers a vote request, as Python you can run. The me object holds this server's saved state, and req is the candidate's request. The list log_terms holds the term of every entry in this server's log, in order.

from types import SimpleNamespace as Msg

def handle_vote_request(me, req):
    if req.term < me.current_term:
        return False                        # candidate is from an old term
    if req.term > me.current_term:
        me.current_term = req.term          # newer term: forget old vote
        me.voted_for = None
        me.role = "follower"
        me.save_to_disk()                   # new term must survive a crash
    my_last_term = me.log_terms[-1] if me.log_terms else 0
    my_last_index = len(me.log_terms)
    up_to_date = (req.last_log_term > my_last_term or
                  (req.last_log_term == my_last_term and
                   req.last_log_index >= my_last_index))
    if me.voted_for in (None, req.candidate_id) and up_to_date:
        me.voted_for = req.candidate_id
        me.save_to_disk()                   # vote must survive a crash
        return True
    return False

class Server(Msg):
    def save_to_disk(self):
        pass                                # a real server writes and syncs to disk here

me = Server(current_term=7, voted_for=None, role="follower", log_terms=[6, 6, 7, 7])
s3 = Msg(term=8, candidate_id="s3", last_log_term=7, last_log_index=4)
s5 = Msg(term=8, candidate_id="s5", last_log_term=7, last_log_index=4)
s1 = Msg(term=9, candidate_id="s1", last_log_term=6, last_log_index=9)
print(handle_vote_request(me, s3), handle_vote_request(me, s5), handle_vote_request(me, s1))
# prints: True False False

Run it with Python 3 and it prints True False False. The second request fails because this server already voted for s3 in term 8. The third has a longer log, but its last entry is from term 6, so it is less up to date.

Two parts matter most. The up_to_date check compares the term of the last entry first and the log length second, exactly as the paper defines it. Both save_to_disk calls run before the reply, because the paper requires the term and the vote to reach stable storage before a server answers.

That is what stops a server that crashes and restarts from voting twice in one term.

In Raft log replication, the leader adds each write to its own log, sends it to every follower, and marks it committed once a majority has stored it. Only committed entries are applied to the key-value map and answered.

Committed means Raft guarantees the entry will never be lost or changed, so it is safe to apply. In the five-server store, a new entry from the current leader is committed once at least 3 servers have stored it, the leader included.

The message that carries entries is called AppendEntries. The same message with no entries is the heartbeat. Each one carries the leader's term and its commit index, the highest entry it knows is committed, so followers learn what they may apply.

Here is one write through the store, step by step. Every box says who talks to whom.

A consistency check keeps the logs identical. Each AppendEntries also names the entry just before the new ones, by its index and term. If the follower has no entry at that index with that term, it refuses the new entries.

In the basic algorithm, the leader then steps back one entry and tries again, until it finds the last point where both logs agree. The follower removes any entry that conflicts with the leader's, plus everything after it, and copies the leader's entries.

So a lagging follower gets repaired without any special recovery code. This gives Raft its Log Matching Property: if two logs hold an entry with the same index and term, the logs are identical in every entry up to it.

Finding the commit point in Python

The leader tracks match_index: the highest entry each server is known to store, counting itself. From that it works out the new commit index. The list log_terms works as in the vote code, so entry number i has term log_terms[i - 1].

def new_commit_index(match_index, log_terms, current_term, commit_index):
    # match_index: highest entry each server stores, leader included
    stored = sorted(match_index.values(), reverse=True)
    majority = len(stored) // 2 + 1
    candidate = stored[majority - 1]        # held by at least a majority
    if candidate > commit_index and log_terms[candidate - 1] == current_term:
        return candidate                    # safe to commit up to here
    return commit_index                     # wait for more copies

match = {"s3": 7, "s2": 7, "s4": 6, "s5": 5, "s1": 3}
log_terms = [6, 6, 7, 7, 7, 8, 8]
print(new_commit_index(match, log_terms, 8, 5))   # prints 6
print(new_commit_index(match, log_terms, 9, 5))   # prints 5: entry 6 is from term 8

It prints 6, then 5. With match indexes 7, 7, 6, 5 and 3, the third-highest value is the highest entry a majority holds, so a term 8 leader commits up to entry 6. In the second call the leader is in term 9, but entry 6 is from term 8, so it waits.

That term check looks like a small detail. The engineer section below shows the crash sequence that makes it necessary.

During a network split, only the side holding a majority of servers can elect a leader and commit writes. The smaller side cannot commit any write, so its clients never get a success reply and there is never a second boss.

Replay the fibre cut with Raft in place. Say s1 was repaired and is leading again by term 11. The cut leaves s1 and s2 on one side, and s3, s4 and s5 on the other. Watch which side can still commit.

Look at the old leader, s1. It can still add a write to its own log, but it can never collect 3 copies. So the write never commits and the client never hears success.

When the network heals, s1 sees term 12 and steps down. Entries it added during the split conflict with the new leader's log, so they are removed or replaced. The client never got a success reply, so it should retry.

A retry could apply a command twice if the first attempt did commit. The paper's answer is for clients to give every command a unique serial number, so the state machine can skip one it has already run.

A Raft cluster needs a majority of its servers alive, so most teams run 3 or 5 servers. Three servers survive one failure, and five survive two.

Majority, also called quorum, means more than half: 2 of 3, 3 of 5, 4 of 7. Adding a fourth server to three raises the quorum to 3 without surviving any extra failure. That is why the etcd FAQ recommends an odd number of members.

More servers is not always better. With seven servers, each write needs 4 stored copies: the leader's own plus 3 follower replies. The etcd FAQ says a cluster should probably have no more than seven nodes, because writes must be copied to more machines.

The team places its five servers across three zones: two, two and one. Losing any whole zone still leaves at least 3 servers, which is a majority.

Raft runs inside systems such as etcd, TiKV and CockroachDB, a distributed SQL database. Apache Kafka, a system for streaming event logs, uses it for its metadata, the cluster's own records about its servers and settings. You rarely call Raft directly. It sits underneath a database or config store.

etcd is a key-value store built on Raft. The etcd project's Go library for Raft says it powers systems such as etcd, Kubernetes, CockroachDB and TiDB. Kubernetes runs containers across many machines, and TiDB is another distributed SQL database.

That library implements only the core algorithm, and you bring your own network transport and disk storage. TiKV, a distributed key-value database, runs Raft through raft-rs, a Rust Raft library kept under the TiKV project on GitHub.

Kafka is another example. Older Kafka clusters kept their metadata in ZooKeeper, a separate coordination service. A proposal called KIP-500 replaced it with a metadata log managed by a Raft quorum of controller nodes.

Kafka 4.0, released on 18 March 2025, is the first major release that runs entirely without ZooKeeper. The leader of the metadata log is called the active controller, and the other controllers stand by to take over.

Kafka's version, called KRaft, is a Raft dialect rather than a copy. Its design proposal, KIP-595, says it is pull-based: followers fetch new entries from the leader, while published Raft has the leader push them.

Raft keeps committed data safe with two rules. Only a server with an up-to-date log can win an election, and a leader only counts copies for entries from its own term.

The first rule works because majorities overlap. A committed entry sits on at least 3 of the 5 servers, and a winning candidate needs 3 votes. Any two groups of 3 out of 5 share at least one server.

That shared server holds the committed entry. It refuses to vote for a candidate whose log is less up to date than its own, so a candidate missing that entry cannot win. Every new leader therefore already holds every committed entry.

Pause and predict. A new leader finds that an entry from an older term already sits on 3 of 5 servers. Is it safe to apply? The answer is no, and this sequence from the paper shows why.

1. 1

   Term 2: s1 leads

   s1 writes entry 2 in term 2 and copies it to s2 only. Then s1 crashes.

2. 2

   Term 3: s5 leads

   s5 wins with votes from s3, s4 and itself, since none of them has entry 2. It writes a different entry 2 in term 3, then crashes before copying it.

3. 3

   Term 4: s1 is back

   s1 wins term 4 and copies its term 2 entry to s3. That entry now sits on s1, s2 and s3, a majority.

4. 4

   s1 crashes again

   s5 can still win term 5, because its last entry is from term 3, newer than term 2. Votes from s2, s3 and s4 are enough.

5. 5

   The majority copy is overwritten

   s5 copies its own entry 2 to the others and replaces the term 2 entry. If s1 had applied that entry in term 4, the servers would now disagree.

This is Figure 8 of the Raft paper, and the fix is the term check in the Python above. A leader only commits an entry from its own term, and that commits every earlier entry with it.

If s1 had stored a term 4 entry on a majority, s5 could never win again, because those servers' logs would end in term 4. The paper also has each new leader commit a blank no-op entry, one that changes nothing, at the start of its term. That tells the leader which entries are committed, and by the rule above it commits every earlier entry without waiting for a client write.

Once the no-op commits, the leader knows exactly which earlier entries are committed. Replaying the sequence with the two Python functions above gives the same result: entry 2 sits on 3 of 5 servers, yet the commit index stays at 1.

An old leader returns stale reads when it answers from its own copy without checking that it is still leader. If a newer leader has already taken writes, that copy is out of date.

The paper's fix has two parts. A new leader first commits its no-op entry, so it knows which entries are committed. Then it exchanges heartbeats with a majority before it answers a read. Ongaro's PhD dissertation spells out the steps, built around a saved commit index it calls the readIndex. It also notes stale-read bugs found in two third-party Raft implementations.

The dissertation also describes leases, which skip that round trip but assume a bound on how far server clocks drift. Use them only when you trust your clocks.

etcd lets you choose per request. Its API guarantees page says it is linearizable by default, meaning a read reflects every write that finished before it began. A serializable read skips consensus and may return stale data.

We checked this on a local five-member etcd 3.4.30 cluster with only two members left. A normal get failed at our 3-second limit, but get with --consistency=s answered at once from the member's own copy. That copy would be stale if a majority elsewhere had taken newer writes.

Pre-vote stops a server that was cut off from forcing a needless election when it returns. Without it, a cut-off server keeps timing out and raising its term while it cannot win.

When it rejoins, that higher term reaches the cluster and forces the healthy leader to step down. Ongaro's dissertation describes this and adds a Pre-Vote phase: a candidate raises its term only after a majority says it would vote for it.

Our etcd 3.4.30 test shows the term growth behind this. We stopped three of five members for about 10 seconds, so the other two could not elect a leader. With pre-vote off, the cluster came back at term 11; with it on, at term 3.

etcd turned pre-vote on by default in v3.5.0. The Ubuntu 24.04 package we tested is still 3.4.30 as of 19 October 2026, and there it is off, so check etcd --version and the --pre-vote flag on older installs.

On an etcd cluster, etcdctl, etcd's command-line tool, shows each member's view of Raft when you run etcdctl endpoint status -w table. It lists whether the member is leader, its Raft term and its Raft index, the highest log entry it knows is committed.

Here is the drill we ran on a local five-member etcd 3.4.30 cluster. Run it only on a test cluster you are allowed to break, never on production.

1. 1

   Read the status

   Run etcdctl --endpoints=ENDPOINTS endpoint status -w table. Exactly one row should say true under IS LEADER, and every row should show the same RAFT TERM.

2. 2

   Stop the leader

   Kill the leader's process and run the command against the other four. In our run, a new leader showed up after about 1.5 seconds, and RAFT TERM rose from 2 to 3.

3. 3

   Stop two more

   With only 2 of 5 members left, etcdctl put fails with context deadline exceeded. The ERRORS column shows etcdserver: no leader. While there is no leader, RAFT TERM can lag. Ours still read 3 while a member's own log already showed term 8, so trust the log.

4. 4

   Bring one back

   With 3 of 5 running, writes work again, and every running member shows the same new term. In our run the term jumped from 3 to 11, because pre-vote is off by default in 3.4.30. The pre-vote section above explains why.

5. 5

   Pause, then resume a leader

   On a fresh cluster started with --pre-vote=true, we froze the leader for 4 seconds with kill -STOP, which pauses a process without ending it. The others elected a new leader in term 3. When the old leader resumed, its log showed the line became follower at term 3.

The 1.5 seconds includes our polling loop. etcd's Raft code picks each election timeout at random between one and two times the setting, so 1 to 2 seconds with the 1,000 millisecond default. Your numbers depend on your settings and network.

Raft's main limits are that every write goes through one leader, every write waits for a majority round trip, and writes stop when no majority can talk.

• One leader handles all writes for a Raft group. That suits small, important data like config and metadata. Large databases split their data instead: CockroachDB divides its keys into ranges, contiguous chunks of keys, and runs one Raft group per range.
• Every commit waits for the slowest server in the fastest majority. Spread servers across far-apart regions and every write pays that round trip.
• Losing a majority stops writes. In a split, Raft keeps the minority side correct but unavailable. The CAP theorem says a split system must choose between consistency, always-current answers, and availability, answering every request. Raft chooses consistency.
• The log grows forever unless you trim it. Servers save a snapshot of the state machine and drop the entries it covers, and a far-behind follower gets the snapshot instead.
• Changing members needs care. The paper uses joint consensus, where old and new majorities must both agree, and the dissertation later recommends adding or removing one server at a time.

In core Raft, timing affects availability, never safety. Lease-based reads are the exception, since they trust clocks. The paper's rule is that broadcast time must be much shorter than the election timeout, which must be much shorter than the average time between one server's failures.

The paper puts broadcast time at 0.5 to 20 milliseconds, depending on storage. The team's store took about 4 milliseconds per write inside one zone and 9 across zones (made up). That is fine for config that changes a few hundred times a minute.

Explain Raft as a replicated log with one elected leader, then walk through the write path and the failure path on a five-server example.

1. 1

   Start with the problem

   Name split brain: a backup cannot tell a dead primary from an unreachable one. Say you need one agreed order of writes.

2. 2

   Introduce the replicated log

   Every server applies the same log in the same order, so every copy matches. Consensus keeps that log identical everywhere.

3. 3

   Walk through an election

   Followers wait a random timeout. A candidate raises the term and asks for votes, and a majority makes it leader.

4. 4

   Walk through a write

   The leader appends, sends AppendEntries, commits at a majority, applies the entry, then answers the client.

5. 5

   Break it on purpose

   Kill the leader, then split the network. Show that the minority side cannot commit and the new leader already holds every committed entry.

6. 6

   Show the expert details

   Mention that leaders only count copies for their own term, that reads need a majority check or a lease, and that pre-vote stops needless elections.

7. 7

   State the costs

   One leader per group, a majority round trip per write, and no writes without a majority. Say when you would pick something else.