Leader Election, Leases and Distributed Locks

The job runs once on every server. Each copy of your app has the same timer, and nothing tells the copies that one of them already started.

Picture a common setup. A company sells warehouse software, and its app runs on three servers so one crash cannot take it down. The setup is made up, but the problem is real.

Every night at 2 a.m., two jobs run. A stock job matches the day's barcode scans against stock records and rebuilds a report. An invoice job bills each of about 600 clients and queues an email.

The team copied the same setup to all three servers, timer included. At the next month-end, each client got three identical invoices: 1,800 emails instead of 600 (made up).

The team needs one rule: whichever server starts the job first, the others stay out. That rule is a distributed lock, and this guide builds one, breaks it and then makes it safe.

A distributed lock is a flag, kept in a shared service, that lets only one process do a piece of work at a time. Every other process sees the flag and waits or skips the work.

You may know locks inside one program. A thread takes a mutex (a lock only one thread can hold), does its work and lets go. The language runtime and operating system enforce it, because all threads share one memory.

Across machines there is no shared memory. So the lock must live in a service every server can reach, such as Redis, ZooKeeper, etcd or a database.

The Redis documentation lists the promises a lock needs. Safety, or mutual exclusion, means at most one client holds the lock at any moment. Liveness means the lock can still be taken eventually, even if the holder crashes.

Those two promises pull against each other. To free the lock after a crash, the service must decide the holder is dead. From outside, a holder that is merely very slow looks exactly like a dead one.

First ask what a double run would cost

Martin Kleppmann, a researcher who writes about distributed systems, splits locks into two kinds by one question. What happens if two processes hold the lock at the same time?

• Efficiency lock. A double run only wastes work. The stock job just rebuilds a report, so a second run burns some CPU and nothing more.
• Correctness lock. A double run breaks data or hurts people. The invoice job records money owed, so a second run means duplicate bills.

An efficiency lock can be simple and occasionally wrong. A correctness lock needs an extra guard, called a fencing token, which this guide builds later.

A lease is a lock that ends on its own after a fixed time unless the holder renews it. The timeout frees the lock when the holder crashes, instead of blocking the job forever.

Think of a kavi sammelan, a poetry evening with many poets and one microphone. The anchor hands the mic to one poet for an eight-minute slot. When the slot ends, the anchor may call the next poet, finished or not.

Here is how the picture maps. The mic is the lock, the anchor is the lock service, the slot is the lease, and a poet asking for two more minutes is a renewal.

The anchor never needs to know whether the poet fainted, wandered off or is still mid-verse. The clock decides, and the evening keeps moving. That is liveness.

Where the picture breaks: a poet sees the anchor's signal, but a frozen server sees nothing. When it wakes, it does not know its slot has ended, and it keeps reading aloud.

So choosing the length is a trade-off. A long lease means a long wait after a real crash. A short lease means a slow holder loses the lock more often, so long jobs take a short lease and renew it.

Use one Redis command that creates a key only if it does not exist yet, with an expiry time. Whichever server creates the key holds the lock.

Redis is an in-memory data store that many teams already run as a cache. The command is SET with two options: NX means set the key only if it is new, and PX gives the expiry in milliseconds.

Each server stores a random value in the key, unique to this attempt. When it finishes, it deletes the key only if the value is still its own. Here is the team's version, using the redis-py client library.

import uuid
import redis

r = redis.Redis(host="lock-redis", port=6379)
RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1] then
    return redis.call('del', KEYS[1])
end
return 0
"""

def run_nightly_invoices():
    my_id = str(uuid.uuid4())                  # unique to this attempt
    if not r.set("lock:invoices", my_id, nx=True, px=30_000):
        return                                 # someone else holds it
    try:
        generate_invoices()                    # the real work
    finally:
        r.eval(RELEASE, 1, "lock:invoices", my_id)

The set call with nx=True is the whole lock. In our test, three clients raced for the same key 500 times, and exactly one won every round on Redis 7.0.15.

The px value is a 30-second lease. The invoice job runs for about 40 minutes at month-end (made up), so a helper thread renews the lease every 10 seconds. It runs the same check as RELEASE, with PEXPIRE (reset the expiry) in place of DEL.

RELEASE is a Lua script, a small program that Redis runs as one step. The Redis documentation says scripts run atomically, so nobody can sneak in between the check and the delete.

You can check the lock by hand with redis-cli, the Redis command-line tool. The second SET gets (nil) because the key exists. PTTL shows the milliseconds left, just under 900000, so your number will differ slightly.

$ redis-cli SET lock:invoices test-1 NX PX 900000
OK
$ redis-cli SET lock:invoices test-2 NX PX 900000
(nil)
$ redis-cli PTTL lock:invoices
(integer) 899927
$ redis-cli DEL lock:invoices
(integer) 1

Newer Redis has a shortcut. The Redis documentation says DELEX, added in Redis Open Source 8.4.0, deletes a key only if its value matches. So DELEX key IFEQ value replaces the script (as of October 2026).

What a Redis replica does to your lock

Many setups add a replica, a copy that takes over if the main node dies. Redis replication is asynchronous by default, so the replica receives each write a moment later, not at once.

The Redis documentation spells out the race. The main node grants a lock and crashes before the replica hears about it. The replica is promoted, and a second client takes the same lock.

Kleppmann's advice for efficiency locks is a single Redis node, accepting that it is sometimes wrong. The team's stock job fits that exactly.

A lease ends by the clock, but the old holder may only be frozen and not know its time is up. When it wakes, it carries on writing as if it still held the lock.

Here is how it plays out on a month-end night (times made up). At 2:14:05 the virtual machine running server 1 is suspended by its host for 40 seconds. No code runs, so no renewal is sent.

The lease runs out about 30 seconds after the last renewal. Every server's scheduler retries the job until tonight's run is marked done. Server 2 retries, finds no lock and starts billing from client 1.

At 2:14:45 server 1 wakes. Its helper thread learns on the next renewal that the lock is gone. Nothing tells the billing loop, so both servers bill the same clients.

The obvious fix is to check the lease just before each write. Kleppmann explains why that cannot work: the pause can strike after the check and before the write, and no code runs to notice.

Long pauses are not exotic. Kleppmann notes stop-the-world garbage-collection pauses (every thread halted) have sometimes lasted several minutes, and gives other ways a process can stall:

• A page fault, where the program waits while a piece of its memory is read back from disk.
• A CPU shared with many other busy processes, so yours waits its turn.
• A network packet delayed for a long time after it was sent.
• A SIGSTOP signal, which an operator or a tool can send to freeze a process.

Clocks add a second risk. The Redis documentation says Redis does not use a monotonic clock (one that never jumps backwards or forwards) for key expiry, so a wall-clock jump can end a lease early.

A fencing token is a number the lock service hands out with every grant, always higher than the last. The database checks it on every write and rejects any number older than the newest it has seen.

The trick is who does the checking. The frozen server cannot know it is stale, so the check moves to the database, which sees every write in order. Kleppmann notes this needs the storage to take an active role.

Back to the kavi sammelan. Now the anchor gives each poet a numbered slip with the mic, and the sound engineer ignores anyone holding a slip older than the newest one seen. The confused poet can talk, but the speakers stay off.

For the team, the invoices database plays the sound engineer. A one-row table called job_fence stores the newest token for the job. The invoices table makes client and month a primary key, a set of columns no two rows can share. So each client gets one invoice per month.

CREATE TABLE job_fence (
    job        text PRIMARY KEY,
    last_token bigint NOT NULL
);
INSERT INTO job_fence VALUES ('invoices', 0);

CREATE TABLE invoices (
    client_id int,
    month     text,
    amount    numeric(12,2),
    token     bigint NOT NULL,
    PRIMARY KEY (client_id, month)
);

Every invoice write goes through this code, written with psycopg 3, the PostgreSQL driver for Python. The token comes from the lock service when the server takes the lock.

import psycopg

class StaleLockError(Exception):
    pass

FENCE = (
    "UPDATE job_fence SET last_token = %s "
    "WHERE job = 'invoices' AND last_token <= %s"
)

def check_fence(conn, token):
    cur = conn.execute(FENCE, (token, token))
    if cur.rowcount == 0:
        raise StaleLockError(f"token {token} is older than the newest")

def save_invoice(conn, token, client_id, month, amount):
    with conn.transaction():
        check_fence(conn, token)
        conn.execute(
            "INSERT INTO invoices (client_id, month, amount, token) "
            "VALUES (%s, %s, %s, %s) "
            "ON CONFLICT (client_id, month) DO NOTHING",
            (client_id, month, amount, token),
        )

def run_invoice_job(conn, token, month, bills):
    with conn.transaction():
        check_fence(conn, token)   # first step: raise the fence
    for client_id, amount in bills:
        save_invoice(conn, token, client_id, month, amount)

check_fence raises the stored number to your token, but only if your token is at least as new. It uses at least, not greater than, because one holder writes hundreds of invoices with the same token.

The fence update and the insert share one transaction, a group of database steps that are saved together or not at all. So a rejected token never leaves an invoice behind.

ON CONFLICT DO NOTHING makes a redo harmless. If the new holder reaches a client the old holder already billed, the insert quietly does nothing instead of creating a second bill.

Where does the token come from?

The token must come from something that never hands out a smaller number after a bigger one, even when servers fail. A counter in one Redis node can lose its newest value when a replica takes over.

Services built on consensus give you such numbers. Consensus means a group of servers agrees on every change even when a minority of them fail.

Kleppmann suggests ZooKeeper's zxid, a number stamped on every change, or a node's version. The etcd documentation says the authors treat etcd's revision number, a counter that rises with every change, as the fencing token.

Redlock, a multi-node Redis algorithm covered later, gives no fencing token. Kleppmann points out that its random value never goes up, so storage cannot tell old from new.

Leader election picks one server out of many to do a job for as long as it stays healthy. In ZooKeeper, each server creates a numbered node, and the server holding the smallest number leads.

Think of leader election as a long-lived lock. Instead of locking each job, the team wants one of the three servers to run the scheduler for every nightly job. If that server dies, another takes over.

ZooKeeper is a coordination service that keeps a small tree of data, made of nodes called znodes, on a group of servers called an ensemble. Its overview page says the service stays available while a majority of those servers are up. etcd, covered below, reaches agreement with Raft, a well-known consensus algorithm.

Two ZooKeeper features make election work. An ephemeral node exists only while the session that created it is alive. A sequential node gets a 10-digit counter added to its name, like n_0000000000.

A session is the link between one client and ZooKeeper, kept alive by small regular PING messages. If the cluster hears nothing within the session timeout, the session expires and its ephemeral nodes are deleted.

1. 1

   Volunteer

   Each server creates an ephemeral, sequential node under the same election node. Server A gets n_0000000000, server B gets n_0000000001 and server C gets n_0000000002.

2. 2

   Find the smallest

   Each server lists the election node's children. The one holding the smallest number, server A, becomes the leader and starts the nightly scheduler.

3. 3

   Watch only the one in front

   Every other server sets a watch, a one-time alert, on the node just before its own. Server B watches A's node, and server C watches B's.

4. 4

   Take over when alerted

   If server A's session expires, its node disappears and only server B is alerted. B lists the children again, sees it is now smallest, and leads.

If every server watched the leader, one failure would wake them all at once. The ZooKeeper recipes page calls this the herd effect, and its lock recipe avoids it the same way.

The election still needs fencing. The ZooKeeper programmer's guide says the cluster, not the client, expires sessions, and a cut-off client is not told until it reconnects.

etcd is the key-value store where the Kubernetes API server keeps cluster state. Kubernetes runs containers across many servers. etcd offers the same building blocks under other names: a lease is a timer you attach keys to, and its lock is tied to a lease.

The etcd documentation is blunt about limits. It says etcd's lock by itself cannot protect resources outside etcd. Those resources must check a version themselves, which is exactly what job_fence does.

Use a single Redis lock when a rare double run only wastes work. Use ZooKeeper or etcd with fencing tokens when a double run would corrupt data. Redlock sits in between and is disputed.

Redlock is described in the Redis documentation. A client tries to take the same lock on several independent Redis nodes, five in the example. The lock counts only if a majority, at least three, agreed in time.

Kleppmann argued in 2016 that Redlock's safety depends on timing assumptions: bounded network delay, bounded pauses and bounded clock error. He advised a consensus system such as ZooKeeper for correctness.

Salvatore Sanfilippo, who designed Redlock, replied the same year with a post asking whether Redlock is safe. He argued the random value can act like a token if the storage does a check-and-set on it.

Both sides agree on one thing: the storage must check something. As of October 2026, the Redis documentation itself tells readers to implement fencing tokens, especially for work that can take a long time.

The team's rule fits in one small picture. Start at the top for each job.

Without fencing, the frozen server woke and kept billing: 6 more clients in the Redis run, and 7 to 10 in the etcd runs. With fencing, its very next write was rejected, in 6 of 6 etcd runs. No client was billed twice there, but the primary key on client and month would have blocked duplicate rows anyway. So the key stops duplicate rows, and the fence stops the woken server writing anything more.

We ran it in October 2026 on one machine: Redis 7.0.15 with redis-py 8.1.0, etcd 3.4.30, PostgreSQL 16.15 with psycopg 3.3.6, and Python 3.13.16. Runner 1 bills 10 clients, one every half second.

To copy a frozen virtual machine, we sent SIGSTOP to the lock holder for longer than its lease, then started a second runner. With etcd we froze both the job and etcdctl, the etcd command-line tool holding the lock.

The etcd lease ends 5 seconds after its last renewal. That renewal came shortly before the freeze, so takeover took under 6 seconds.

In our first fenced run the tokens were 9 and 11, not 9 and 10. Revisions rise with every change in etcd, so tokens skip numbers. That is fine, because fencing only needs a bigger number, not the next one.

You can get the token yourself. etcdctl lock --ttl=5 invoices -- ./job.sh runs job.sh while holding the lock. The script sees the token in the ETCD_LOCK_REV environment variable, the etcd revision when the lock was granted. We ran 3.4.30, an older release line, and the etcdctl source for 3.7.2 sets the same variable.

1. 1

   Start the servers

   Run a scratch Redis, etcd and PostgreSQL on your laptop. Create job_fence and invoices with the SQL above.

2. 2

   Start runner 1

   Run the job under etcdctl lock with a short TTL, such as 5 seconds, and let it bill a few clients.

3. 3

   Freeze it

   Send SIGSTOP to runner 1's whole process group, so the job and etcdctl both stop.

4. 4

   Start runner 2

   It should get the lock within about one TTL of the freeze, with a higher ETCD_LOCK_REV.

5. 5

   Wake runner 1

   Send SIGCONT. Expect its next save_invoice to raise StaleLockError. Then SELECT client_id FROM invoices GROUP BY client_id HAVING count(*) > 1 should return no rows.

Fencing only protects writes that pass through a store that checks the token. We found three gaps in testing and in the docs, and each has a simple fix.

A stale write before the new holder's first write

The database learns a new token only when the new holder first writes. Until then, the old holder's token is still the newest it has seen.

In our PostgreSQL test, token 41 wrote successfully while the next holder, token 42, had not touched the fence yet. Once 42 raised the fence, 41 was rejected.

That is why run_invoice_job calls check_fence as its very first step, before billing anyone. The gap shrinks to the moment between taking the lock and that first call.

Side effects outside the database

An email service or a payment gateway does not check your tokens. If the job sends the email itself, a frozen holder can still send one after waking.

So the team's job only writes invoice rows, and a separate mailer sends one email per new row. The fenced database decides what exists, and the email follows it. The mailer needs one runner of its own, or the double send just moves there.

Counters that restart

ZooKeeper's programmer's guide says the sequence counter is kept by the parent node. In our test on ZooKeeper 3.9.1 with kazoo 2.11.0, deleting and re-creating the election node restarted names at n_0000000000.

The node's czxid, the zxid of the change that created it, kept rising: 6, 8 and 10 before, 18 after. So use czxid as the token, or never delete the election node.

One more ZooKeeper result is worth knowing. We froze the leader for 12 seconds with a 4-second session timeout. Only the next server in line was alerted, about 4.7 seconds later.

The frozen leader heard nothing until it woke. Its client then reported the connection as suspended, and as lost about 0.1 seconds later. For that moment it still believed it led, which is the gap fencing covers.

Start from why only one runner matters, show why a lease alone is not enough, and finish with fencing tokens and idempotent writes, meaning writes that are safe to repeat.

1. 1

   Say why you need one runner

   Name the job and what a double run costs. Then say whether this lock is for efficiency or for correctness.

2. 2

   Pick a lock service

   For efficiency, a single Redis key with NX and an expiry. For correctness, ZooKeeper or etcd, which agree on changes by majority.

3. 3

   Add a lease and renewal

   Explain the trade-off: long leases slow recovery, short ones cause false takeovers. Long jobs renew a short lease in the background.

4. 4

   Show the pause problem

   Draw a holder that freezes past its lease while a second holder starts. Say why checking the lease before each write cannot fix it.

5. 5

   Add fencing tokens

   The lock service hands out a rising number. The database rejects any write with a smaller number than it has already seen.

6. 6

   Make the work idempotent

   Give each unit of work a unique key, such as client and month, so a redo by the new holder cannot create a duplicate.