How ZooKeeper Works: Znodes, Watches and ZAB

The same work runs twice. When two copies of a program each decide on their own that they lead, nothing outside them settles who really does. ZooKeeper exists to be that outside referee.

Picture a common setup. A bank's nightly batch scheduler posts interest, builds statements and sends a settlement file. The team runs two copies on two machines, so one can take over if the other dies. The setup is made up, but the problem is real.

A flag in the shared database says which copy is active, and each copy checks it every 30 seconds. One night the network between the active copy and the database freezes for two minutes. The standby sees no update, decides the active copy is dead, and takes over.

The first copy was never dead, only cut off. When the network comes back, both copies run the interest job, and about 1,800 accounts get their monthly interest twice. Each copy guessed who was in charge from its own partial view.

The fix is a referee that both copies trust more than their own guesses. This guide builds that referee with ZooKeeper, one piece at a time, on the same scheduler.

ZooKeeper is an open-source coordination service from the Apache Software Foundation. A small group of servers keeps one shared tree of tiny records, and your programs read, write and watch that tree to agree on facts.

Coordination means getting many machines to agree on small facts. Who is the leader? Which workers are alive? What is tonight's batch window? Every machine must see the same answer, or you get the double run above.

The ZooKeeper servers together are called an ensemble. Each one keeps the whole tree in memory, which is why reads are quick. The official overview says ZooKeeper performs best when reads outnumber writes, at ratios around 10 to 1.

Picture the line-up board in a hockey team's dressing room. Nobody argues about who starts from memory. Everyone checks the board. ZooKeeper is that board for your servers, and the parts map like this:

• The board is the znode tree: one shared place that everyone reads.
• Each name on the board is a znode, one small record.
• A player who asks to be tapped on the shoulder when the board changes has set a watch.
• Only the captain writes on the board. In ZooKeeper, one leader server puts every write in order.

Where the picture breaks: a real board is one object in one room. ZooKeeper keeps a copy of the board on every server, and the copy you read can be a moment behind. That gap gets its own section later.

ZooKeeper does not run your jobs, store your bank data or carry messages. It holds the few shared facts that let your own programs stop guessing. These are its common uses:

• Leader election: picking exactly one active scheduler, and a new one when it dies.
• Group membership: keeping a live list of which workers are up right now.
• Configuration: holding settings that every machine must read the same way.
• Locks and barriers: making programs take turns, or wait at a barrier until all of them are ready.

A znode is one record in ZooKeeper's tree. It has a path like /scheduler/config, a small piece of data, and a stat block: metadata such as version numbers. A znode can also have children, like a folder that holds files.

Here is the scheduler's tree. Look at the three branches under /scheduler. Each one solves a different coordination problem, and the note on each box says how long that znode lives.

Every client connects with a session, a long-lived link between one program and the ensemble. The client keeps it alive with small heartbeats. If the ensemble hears nothing for the session timeout, the ensemble, not the client, expires the session.

An ephemeral znode belongs to one session and is deleted when that session ends. So a crashed worker drops out of /scheduler/workers without anyone cleaning up. In our test, a frozen worker's znode vanished about 7 seconds after it froze, with a 6-second timeout.

A sequential znode gets a counter added to its name when it is created. The counter is ten digits with leading zeros, and the parent znode keeps it. That is how the two candidates above got the numbers 41 and 42.

Each znode also carries a data version that goes up by one on every change. A write can name the version it expects. If someone else changed the znode first, the write fails instead of silently overwriting their change.

A watch is a one-time alert on a znode. You set it while reading, and ZooKeeper sends one notice when that znode's data, existence or children change. After that notice, the watch is gone.

You set a classic watch with one of three read calls: exists, getData or getChildren. The notice says only that something changed, not the new value. You read the znode again to see it, and set a fresh watch in that same read.

We checked each kind on ZooKeeper 3.9.5:

• A data watch, set by getData or exists, fired once for two quick writes. It also fires when the znode is deleted.
• An exists watch on a missing znode fired when the znode was created.
• A child watch, set by getChildren, fired when a child was added. It stayed silent when a child's data changed.
• Since 3.6.0, addWatch sets a persistent watch that stays after it fires. Its recursive mode reported a create, two data changes and a nested create under one path.

ZooKeeper also promises order. A client sees the watch notice before it can see the new data that caused it. When a client reconnects, its watches are registered again and fire if needed.

The scheduler uses a watch so every copy picks up a new batch window without a restart. It uses kazoo, a Python client for ZooKeeper. Its DataWatch helper calls your function once at the start, then again after every notice, setting the watch each time.

from kazoo.client import KazooClient

zk = KazooClient(hosts="zk1:2181,zk2:2181,zk3:2181")
zk.start()

@zk.DataWatch("/scheduler/config/batch-window")
def on_window_change(data, stat):
    if data is None:
        print("Window znode missing, keep the old window")
        return
    print("Batch window is now", data.decode("utf-8"), "version", stat.version)

input("Watching. Press Enter to stop.\n")
zk.stop()

The hosts string lists all three servers, so the client can move to another one if its server fails. When the znode does not exist, kazoo passes None for both values, so the code keeps its old setting.

To check it, start the script against a fresh znode holding 01:00-04:00. In zkCli.sh, run set /scheduler/config/batch-window 01:30-04:00. Within a second the script prints Batch window is now 01:30-04:00 version 1.

Each candidate creates an ephemeral sequential znode under one election path, and the lowest number leads. When the leader dies, its session expires, its znode vanishes, and the next number takes over.

The trick is in who watches whom. Each candidate watches only the znode just below its own number, not the leader's. The official recipe does this to avoid the herd effect: every waiting client waking at once to chase one change.

Follow the diagram from top to bottom. Copy B is the candidate here, and the question box is the check it runs every time its watch fires.

Here is what happens on a bad night with the new design.

1. 1

   Copy A leads

   Copy A holds n_41, the lowest number, so it runs the interest job. Copy B holds n_42 and watches n_41.

2. 2

   Copy A goes quiet

   The network to Copy A freezes. Its heartbeats stop reaching the ensemble.

3. 3

   The session expires

   After the session timeout, the ensemble expires Copy A's session and deletes n_41. Nobody had to guess.

4. 4

   Copy B takes over

   Copy B's watch fires. It lists the children, sees n_42 is now lowest, and becomes the leader.

5. 5

   Copy A must stand down

   When Copy A's network returns, its client learns the session is gone. Its code must stop leading at once and rejoin as a fresh candidate.

Kazoo wraps this recipe in an Election object, built on its lock recipe, which also watches only the next lower znode. Its run call blocks until this copy wins, then calls your function. It gives up leadership when that function returns.

It wins at once. In our test of that pattern, the standby became leader 0.6 seconds later, while the first copy's jobs were still running. That is the double run again, now caused by the fix.

So the leader function must keep running for as long as this copy should lead. In the code below, run_due_jobs is a stand-in for the scheduler's own function that starts any jobs that are due. Beginners can skim the code; the paragraph after it explains it.

import threading
import time
from kazoo.client import KazooClient, KazooState

zk = KazooClient(hosts="zk1:2181,zk2:2181,zk3:2181")
healthy = threading.Event()   # set while the connection is up
lost = threading.Event()      # set once the session has expired

def on_state(state):          # runs on every connection change; never block here
    if state == KazooState.LOST:
        lost.set()
        healthy.clear()
    elif state == KazooState.SUSPENDED:
        healthy.clear()
    else:
        healthy.set()

zk.add_listener(on_state)
zk.start()
election = zk.Election("/scheduler/election", identifier="sched-a")

def run_as_leader():
    me = election.lock.path + "/" + election.lock.node
    token = zk.exists(me).czxid             # id of the write that created our candidate
    while not lost.is_set():
        if healthy.is_set():
            run_due_jobs(token)             # your code: start jobs, pass the token
        time.sleep(5)
    # returning ends this term and gives up the election znode

while True:
    lost.clear()
    election.run(run_as_leader)   # blocks until this copy wins, then calls the function

What just happened? The leader function loops instead of returning, so this copy keeps its znode. The listener pauses new jobs while the connection is shaky (SUSPENDED) and ends the term once the session has expired (LOST).

The token is the czxid of this copy's candidate znode: ZooKeeper's id for the write that created it. These ids only grow, so each new leader gets a larger token. Kazoo 2.11.0 sets election.lock.node but does not document it, as of October 2026, so pin your kazoo version.

So the listener alone cannot stop a stale leader. The real guard is a fencing token: a number that grows with each new leader, so storage can refuse writes that carry an older one.

Make each job idempotent, meaning safe to repeat. Then have the bank's database reject any job whose token is lower than the last one it accepted. The guide on distributed locks and leader election covers fencing in full.

ZAB, short for ZooKeeper Atomic Broadcast, sends every write through one leader server. The leader numbers each write, followers save it to disk in that order, and it commits once a majority have saved it.

Do not mix up the two leaders. The scheduler's leader is one of your app's copies. ZAB's leader is one of the ZooKeeper servers, chosen by the servers themselves, and your app never has to think about it.

Every change gets a zxid, a 64-bit transaction id. The high 32 bits are the epoch, which goes up each time a new ZooKeeper leader takes over. The low 32 bits count changes within that epoch.

You can watch the epoch move. In our test, a write before we killed the leader had zxid 0x2000003ae, where the leading 2 is the epoch. The first write after a new leader took over was 0x300000001: epoch 3, change 1.

Now follow one write through the ensemble. Copy B is creating its candidate znode and happens to be connected to a follower. Each box says who talks to whom.

The internals page compares this to a two-phase commit, a method that first asks everyone to accept a change and then tells them to apply it. It adds that ZAB never needs to handle aborts, the step where a change is cancelled.

A follower's acknowledgement means it has recorded the proposal to persistent storage. Proposals go out in the order the leader received the requests, and followers commit them in that same order.

Leader changes are where ZAB earns its keep. A new leader first brings followers up to date, sending missing proposals or a full snapshot of the tree. It accepts no new writes until a majority have synced with it.

Any two majorities of the same ensemble share at least one server. A committed write sits on a majority, so the new majority always includes a server that has it. The internals page relies on exactly this overlap.

If Raft is new to you, the Raft consensus guide covers leaders, terms and log matching step by step. Most of it carries over to ZAB.

Use an odd number, usually three or five. An ensemble keeps working while a majority of its servers, called a quorum, is up. Three servers survive one failure, and five survive two.

The rule is 2F plus 1: to survive F failed servers, run 2F plus 1 of them. A sixth server adds nothing to five. Six needs four for a majority, so it still survives only two failures.

The admin guide calls three the minimum for a fault-tolerant setup and suggests five if you want room for maintenance. With five, one server can be down for an upgrade while the ensemble still survives another failure.

We checked the rule on a three-server ensemble. Killing the leader brought in a new one after 0.81 seconds. Killing a second server left the last one refusing to serve, and a fresh client could not connect, so its write failed after 8 seconds.

More servers do not make writes faster. The leader sends every proposal to every follower and waits for a majority, so a bigger ensemble does more work per write. The team runs five servers across three racks, so no single rack holds a majority.

• Spread servers so that one switch, rack or power line cannot take out a majority.
• Put the transaction log on its own device. The admin guide says a dedicated partition is not enough.
• Keep the data small, since every server holds the whole tree in memory.
• Watch for slow followers. The syncLimit setting drops followers that fall too far behind the leader.

No. A read is answered by whichever server your client is connected to, from its own copy. That server may not have applied the latest commit yet, so a read can be slightly behind.

This is a design choice. Answering reads locally is what makes ZooKeeper quick for read-heavy coordination, while every server still applies writes in the same zxid order.

The programmer's guide lists what you can count on instead:

• Sequential consistency: updates from one client are applied in the order that client sent them.
• Atomicity: an update either fully succeeds or fully fails.
• Single system image: a client sees the same view whichever server it connects to.
• Reliability: once an update is applied, it stays until a client overwrites it.
• Timeliness: a client's view is up to date within a time bound, which the guide puts on the order of tens of seconds.

Single system image matters in practice. If the scheduler's client fails over from a fresher server to a lagging one with the same session, it never sees an older view. Its view never goes backwards.

When one client must see another client's latest write, call sync before the read. Sync asks your server to catch up with the leader first. The internals page notes that sync is not currently a quorum operation, so even this is not a hard promise.

We ran a three-server ZooKeeper 3.9.5 ensemble on one machine, with kazoo 2.11.0 and the official Java client. The tickTime, the server's base time unit, was 2,000 ms. Every row below comes from those runs.

Size limits fail like network errors

On the six-character path /t/big, the largest data that fitted was 1,048,549 bytes. Each extra character of path cost one byte of data, so the limit covers the path too. The admin guide describes jute.maxbuffer as the largest data a znode can store, so expect a slightly lower real cap.

The client did not get a clean error. The server dropped the connection, and kazoo reported ConnectionLoss.

If you see repeated connection losses right after a large write, check the payload size before you blame the network. Raising jute.maxbuffer is possible, but the client value should match the server value, so it is better to store a pointer.

Pick a fencing token that only grows

The sequence counter lives in the parent znode and moves on every child create, not just sequential ones. If someone deletes and re-creates /scheduler/election, the counter restarts at zero, and a new leader can carry a lower sequence number.

That is why the code above uses the candidate znode's czxid, the zxid of the write that created it. Zxids grow across the whole ensemble, through every leader change, so a newer leader always holds a larger one.

Cleanup timers are slower than they look

Container and TTL znodes are removed by a background check that runs every 60,000 ms by default. The admin guide names it znode.container.checkIntervalMs, a Java system property new in 3.6.0. A 2-second TTL therefore means gone within about a minute, not gone after 2 seconds.

Kafka dropped ZooKeeper to run one system instead of two. Its 4.0 release, announced on 18 March 2025, was the first major release to operate entirely without ZooKeeper, running in KRaft mode by default. In KRaft mode, some Kafka servers act as controllers that store all the cluster's metadata.

Before KRaft, a Kafka cluster needed a ZooKeeper ensemble beside it. The 4.0 announcement says the change removes the complexity of maintaining that separate ensemble.

ZooKeeper itself is still maintained, with 3.9.6 as its current release as of October 2026. HDFS, Hadoop's distributed file system, still uses it for automatic failover of the NameNode, the server that handles all client operations. Each NameNode keeps a ZooKeeper session, and an ephemeral lock znode decides which one is active.

ZooKeeper fits when several separate programs need shared, ordered facts and you are willing to run an ensemble. The scheduler fits that well. Look elsewhere in these cases:

• You need to store real data, like files, logs or customer records. That belongs in a database or object storage.
• Your workload is mostly writes. ZooKeeper is tuned for read-heavy loads, around 10 reads for every write.
• Your platform already runs a coordination store, so a second one adds work without adding safety.
• You only run one copy of your program. A single process has nobody to coordinate with.

Explain it as a small, replicated tree of facts that your servers trust more than their own guesses. Then walk through one problem it solves, end to end.

1. 1

   Start with the failure

   Describe two copies that each think they lead, like the double interest posting. Say the root cause is that each made its own guess.

2. 2

   Introduce the tree

   Explain znodes, paths and sessions. Show that ephemeral znodes disappear when their session expires, so liveness comes almost for free.

3. 3

   Add watches

   Explain that a watch is a one-time alert, that bursts can merge, and that clients read again after each notice.

4. 4

   Walk through leader election

   Use ephemeral sequential znodes; the lowest number wins, and each candidate watches only the one just below it to avoid the herd effect.

5. 5

   Explain the ensemble

   Writes go through a leader using ZAB and commit on a majority. Use three or five servers, and say reads may be stale without sync.

6. 6

   Close with the limits

   Mention that a paused leader can act late, so jobs need fencing tokens and must be safe to repeat. Note that Kafka now runs without ZooKeeper.