How Google Spanner Works: TrueTime and Global Consistency

When each server stamps events with its own clock, and the clocks disagree, an event that happened later can carry an earlier time. Sorting by timestamp then puts events in the wrong order. Google Spanner is a database built to keep that order right across the world.

Picture a common setup. A remittance team runs a money ledger for workers in the Gulf who send money home to India. Its servers sit in Dubai and Mumbai and handle about 80,000 transfers a day. The setup is made up, but the problem is real.

The ledger is a database table where every movement of dirhams or rupees is one row. A nightly audit job replays the day's rows in timestamp order. It checks that every refund comes after the transfer it refunds.

One night the audit flags 63 refunds that appear before their own transfers. An engineer picks one and traces it by hand.

• A customer in Dubai sends AED 1,000. The Dubai server stamps the transfer 23:10:00.000, and its clock happens to be right.
• Twenty milliseconds later, a server in Mumbai issues a small fee refund for that transfer.
• The Mumbai server's clock is 45 milliseconds slow, so it stamps the refund 23:09:59.975.
• In timestamp order, the refund now sits 25 milliseconds before the transfer it refunds.

Nothing was lost, and every row is correct. Only the order is wrong. For a regulated money ledger, though, the order is part of the truth.

Better syncing with NTP, the Network Time Protocol that servers use to set their clocks, shrinks the error. But each server still hands out one timestamp per row, with no idea how wrong it might be.

Google needed the same guarantee across the whole planet, and Spanner is how it got there. The rest of this guide follows the team's ledger through it.

Google Spanner is a distributed SQL database that keeps copies of your data in many places and still runs transactions in one global order. Google built it for its own services, opened it to Google Cloud customers as a public beta in February 2017, and made it generally available later that year.

Distributed means the data lives on many machines, often in different cities. SQL means you define tables with primary keys and query them with the usual database language.

Google described Spanner in a 2012 paper at OSDI, a leading operating systems conference. The paper calls it the first system to distribute data at global scale and support externally consistent distributed transactions.

Spanner rests on three ideas, and the next sections take them one at a time:

• Splits and Paxos: each table is cut into ranges of rows, and each range is copied to servers that vote on every write.
• TrueTime: a clock service that reports its own uncertainty as a range.
• Commit wait: a short pause before each write becomes visible, which turns that range into a promise about order.

Spanner cuts each table into splits, which are ranges of rows ordered by primary key, and copies each split to several replicas. The replicas of a split form a Paxos group, which elects one replica as leader.

A primary key is the column that identifies each row, like wallet_id in the ledger's wallet table. A replica is one copy of the data, kept on its own server.

Paxos is a consensus protocol: a set of rules that lets several replicas agree on one order of writes, even when some of them fail. A separate guide on Paxos covers how the voting works.

Every write goes to the leader first. The leader logs it and sends it to the other voting replicas. The write commits once a majority of voting replicas, which the docs call the write quorum, agree.

As of October 2026, Spanner's replication docs list three kinds of replica:

• Read-write replicas hold a full copy, serve reads, vote on writes and can become the leader.
• Read-only replicas hold a full copy and serve reads, but do not vote, so they add no delay to writes.
• Witness replicas vote on writes and take part in leader elections, but hold no full copy, serve no reads and never lead.

A leader keeps its role through a lease, a time-limited right to lead that it keeps renewing. In the 2012 paper, leases last 10 seconds by default, and no two leaders of the same split ever hold overlapping leases.

A region is one geographic area, like Mumbai, with its own data centres. The team plans a multi-region layout with the same shape as nam3 and eur3, two layouts the docs list. Two regions hold two read-write replicas each, and a third holds a witness.

One of the two read-write regions is the default leader region, where leaders normally sit. In the diagram that is region A.

Start at the app servers and follow the write to the leader, then out to the voters. TrueTime feeds the leader, because the leader picks each write's timestamp.

If region B goes dark, the leader, replica A2 and the witness still make three of five. If region A goes dark, the three survivors can elect a new leader once the old lease runs out.

Paxos gives each split a safe, ordered history. It does not order writes across different splits, though. Ordering a refund in one split against a transfer in another is TrueTime's job.

TrueTime is Spanner's clock service, and it returns a time range instead of a single time. The paper's promise is that the real current time lies somewhere inside that range.

A normal clock gives you one number, like 23:10:00.000, and says nothing about how wrong it might be. TrueTime's call TT.now() returns two numbers, earliest and latest. The paper calls half the width of that range epsilon, the error bound.

Two helper calls build on it. TT.after(t) is true when time t has surely passed, because even the earliest possible current time is later than t. TT.before(t) is true when t has surely not arrived yet.

How TrueTime is built

Each data centre has a set of time master machines. Most have GPS receivers with their own antennas, placed apart from each other. The rest, which the paper calls Armageddon masters, have atomic clocks.

Spanner uses both kinds because they fail in different ways. GPS can suffer antenna faults, radio interference, spoofing or outages. Atomic clocks fail in ways unrelated to GPS, but can drift over long periods.

Every machine runs a daemon, a small background program, that polls several masters. It uses a variant of Marzullo's algorithm, a method for combining time sources, to spot masters that disagree and reject them as liars.

Each master also compares its time reference with its own local clock, and evicts itself if the two diverge badly. Machines whose clocks drift beyond the worst case the bound allows are evicted too.

How wide the range is

The 2012 paper reports that epsilon moves in a sawtooth between about 1 and 7 milliseconds, and is 4 milliseconds most of the time. Spanner's TrueTime docs page gives no current figure, as of October 2026.

The sawtooth comes from the polling. The daemon polled every 30 seconds. Between polls, it assumed the local clock may drift by up to 200 microseconds every second, so the range slowly widens.

Thirty seconds of that drift adds up to 6 milliseconds, then the next poll snaps epsilon back down. Delay to the masters adds the remaining 1 millisecond.

Commit wait means the leader picks a commit timestamp at the top of the TrueTime range, then holds the result back until that timestamp is surely in the past. Only then can anyone see the write.

A commit timestamp is the time Spanner officially records for a transaction. Spanner orders transactions by these timestamps, and every read uses them to decide which data it sees.

The paper states the rule in two parts. First, the commit timestamp s must be no less than TT.now().latest, measured after the commit request arrives. Second, no client may see the data until TT.after(s) is true.

Here is why that works. Say transaction T1 finishes, and transaction T2 starts afterwards, on any server in the world. Because of the wait, T1's timestamp s1 was already in the past when T1 finished.

T2's timestamp s2 is at least the latest bound when T2's commit request arrives. That arrival comes after T2 started, so s2 is later than T2's real start time. That makes s1 smaller than s2. No shared clock and no message between the two servers was needed.

The parade marshal's rule

Picture the Republic Day parade. A marshal at the start line holds the next contingent until she is sure the previous one has fully passed the saluting dais. Her stopwatch and the announcer's never agree to the second.

So she adds her doubt to the wait. If her stopwatch might be two seconds off, she holds the contingent two seconds longer. Contingents then always pass the dais in order.

The leader is the marshal, the stopwatch is the local clock, and the two seconds of doubt is epsilon. Where the picture breaks: the marshal holds everyone, but Spanner holds back only transactions that lock the same rows.

That is because locks stay held until the wait ends, while waits for unrelated rows run side by side.

Here is a toy version you can run. The tt_now helper fakes TrueTime with a fixed 4 millisecond error. The write_to_replicas argument stands in for the Paxos write to a majority.

import time

EPSILON = 0.004  # assumed clock error: 4 ms either way

def tt_now():
    t = time.time()
    return (t - EPSILON, t + EPSILON)  # (earliest, latest)

def tt_after(ts):
    earliest, _ = tt_now()
    return earliest > ts  # ts has surely passed

def commit(write_to_replicas, change):
    s = tt_now()[1]                # timestamp from the top of the range
    write_to_replicas(change, s)   # stand-in for the Paxos majority write
    while not tt_after(s):         # commit wait
        time.sleep(0.001)
    return s                       # only now is the client told

start = time.time()
s = commit(lambda change, ts: None, "transfer")
print(f"waited {(time.time() - start) * 1000:.1f} ms")
print("timestamp surely past:", tt_after(s))

Two lines carry the whole idea. The timestamp comes from the latest end of the range, never the middle. The while loop is commit wait: it ends only when even the earliest possible time is past s.

To check it, save the code as commit_wait.py and run python3 commit_wait.py on Python 3.8 or later. Expect a first line such as waited 8.7 ms and a second line ending in True.

In 50 runs on Python 3.13, the wait was 8.0 to 13.1 milliseconds, with a median of 8.7. It never drops below 8, which is twice epsilon. Sleep timing adds the rest.

In the paper's 2013 journal version, a one-replica write took 10.1 ms with commit wait turned off and 14.1 ms with it on. The paper notes the wait typically overlaps the Paxos voting, so a write does not pay all of it on top.

Now replay the broken refund on Spanner. The transfer commits at s1, and the app hears about it only after s1 has surely passed. Only then can anyone issue the refund.

The refund's leader picks s2 from the top of its own range, which is later than the real time. So s2 is larger than s1, whatever either server's clock says, as long as each clock stays inside its bound.

We tested that condition in a small simulation with made-up clocks. Across 100,000 transfer-then-refund pairs, none flipped while clock errors stayed inside the bound. With errors three times the bound, 21,762 flipped, even with commit wait.

External consistency means that if one transaction commits before another starts, the second gets a larger commit timestamp and sees the first. The order people observe in the real world is the order the database records.

Spanner's docs say the system behaves as if all transactions ran one after another, even though they run across many servers. They call external consistency stricter than serializability and stronger than linearizability.

Serializability only promises that the result matches some one-by-one order, which may not match real time. Linearizability respects real time, but usually for single operations on one item. A separate guide on consistency models compares these levels.

External consistency combines both: whole transactions, touching many rows in many splits, line up in real-time order. The docs describe linearizability as a special case of it.

The docs give a banking example that fits the ledger well. A deposit into savings is followed by a debit from checking. Every reader sees the deposit first, so the debit never triggers a false insufficient-funds penalty.

Spanner runs two-phase commit across the Paxos groups a transaction touches, with one group's leader as the coordinator. Every other participant prepares first, then the coordinator picks one timestamp, waits, and tells everyone to commit.

Two-phase commit is a protocol where every participant first promises it can commit, called the prepare phase. Only then does a coordinator announce the final decision. In Spanner, each participant is a whole Paxos group, not one machine.

That design answers an old complaint. In classic two-phase commit, a crashed coordinator can leave everyone stuck. The paper says running it over Paxos mitigates these availability problems, because a group survives one machine's crash.

Take one transfer that touches two splits. The wallet split holds the sender's dirham balance, and the payout split records the rupees owed in India. The wallet split's leader is the coordinator. Follow the diagram from top to bottom.

The coordinator's timestamp s must be at least every prepare timestamp, and above its own TT.now().latest when the commit message arrived. It must also be larger than any timestamp it gave out before. Then comes commit wait.

After the wait, the coordinator sends s to the app and to every participant. Each participant logs the outcome through its own Paxos group. All of them apply the writes at the same s, then release their locks.

Reads inside a read-write transaction go to the leader and, by default, take shared read locks. Spanner uses the wound-wait scheme for deadlocks, where an older transaction can abort a younger one.

Interleaving helps here. An interleaved child table, such as transfers, is stored with its parent wallet row. The docs say such rows usually share a split, as long as the parent and its children stay below the split size limit.

Spanner keeps old versions of every row, each tagged with its commit timestamp, so a read can pick a timestamp and see a fixed snapshot without taking locks. Writers keep working while readers read.

This idea is called MVCC, multi-version concurrency control. A snapshot read at time t returns, for every row, the newest version at or before t. Read-only transactions work this way, hold no locks and block no other transactions.

There is one catch. A replica may answer a read at time t only if it is sure it already holds every write up to t. The paper calls this safe time: the highest timestamp up to which a replica is complete.

Two things hold safe time back. One is Paxos writes the replica has not applied yet. The other is a transaction that is prepared but not committed, because it may still commit at or above its prepare timestamp.

The code below is a simplified model of that rule, in made-up whole-number timestamps. The near replica is behind. The far replica is caught up but holds one prepared transfer at timestamp 250.

from dataclasses import dataclass

@dataclass
class Replica:
    name: str
    distance_ms: int
    last_applied_ts: int            # newest Paxos write applied here
    oldest_prepared_ts: int | None  # None if nothing is prepared
    versions: list                  # (commit_ts, balance), oldest first

    def value_at(self, ts):
        older = [v for t, v in self.versions if t <= ts]
        return older[-1] if older else None

def safe_time(r):
    if r.oldest_prepared_ts is None:
        return r.last_applied_ts
    return min(r.last_applied_ts, r.oldest_prepared_ts - 1)

def snapshot_read(replicas, read_ts):
    for r in sorted(replicas, key=lambda r: r.distance_ms):
        if read_ts <= safe_time(r):
            return r.name, r.value_at(read_ts)
    return None, None  # nobody has caught up yet: wait and retry

near = Replica("near", 2, 170, None, [(100, 5000)])
far = Replica("far", 40, 260, 250, [(100, 5000), (180, 4000)])
print(snapshot_read([near, far], 150))
print(snapshot_read([near, far], 200))
print(snapshot_read([near, far], 255))

Run it with Python 3.10 or later. It prints three lines: ('near', 5000), then ('far', 4000), then (None, None).

What just happened? At 150 the nearby replica was complete, so it answered. At 200 it had only applied writes up to 170, so the read went to the far replica. At 255 nobody qualified, because the prepared transfer caps far's safe time at 249.

The paper's real rule has more parts, such as leaders promising future timestamps, but the min line is the heart of it. A replica that is behind, or holding a prepared write, never guesses.

Spanner gives you several ways to read. The table lists them as the docs describe them in October 2026, with the job the ledger team gives each one.

The docs say to choose strong reads whenever possible. When slightly old data is fine, they call 15 seconds a reasonable staleness value for good performance.

For a strong read at a non-leader replica, the round trip to the leader only agrees a timestamp. The data itself comes from the nearby replica.

The nightly audit gains the most. It reads the whole ledger at one timestamp, so every row belongs to one consistent moment, and it never blocks the first transfers of the new day.

Four documented details surprise teams that otherwise understand TrueTime. Each one bites the ledger team's audit or schema if nobody plans for it. All are as the docs stand in October 2026.

The tie rule fits external consistency. A refund issued after its transfer committed always gets a larger timestamp. Ties only happen between transactions with no real-time order, so any stable tie-breaker is safe.

Retention is the trap with the biggest blast radius. An audit that reads as of midnight but starts at 2 a.m. fails on a database left at the 1 hour default.

Spanner trades write speed and money for global order and high availability. Every write pays commit wait, plus a majority vote that may cross regions.

The CAP theorem says that during a network partition a system must give up either consistency or availability. A partition is a break that splits servers into isolated groups.

In February 2017, Google's Eric Brewer wrote that Spanner is technically a CP system: during some partitions it keeps consistency and gives up availability. He also reported more than five 9s of availability, meaning under one failure in 100,000. A separate guide covers CAP and PACELC.

In practice, voters cut off from a majority cannot commit writes for their split. Most failures, like one server or one data centre going down, still leave a majority running. The team lists the costs for its managers:

• Write delay: commit wait adds a few milliseconds, and a quorum that crosses regions adds a network round trip on top.
• Leader distance: every write goes to the leader of each split. A strong read away from the leader adds a round trip to agree a timestamp.
• Key design: a key that always increases, like a timestamp, sends new rows to the end of the key space and overloads one server.
• Money: a multi-region layout runs replicas in three regions instead of one, so it costs more than a single-region database.

Spanner is more than you need in some common cases:

• All your users and data sit in one region, and one primary database with a standby copy is enough.
• Slightly out-of-date or out-of-order data is harmless, like view counts or likes.
• Your writes are so frequent and tiny that a few milliseconds each would hurt more than a wrong order.

The team's decision is narrow on purpose. Only the ledger moves to Spanner. Exchange-rate caches, marketing data and chat logs stay in cheaper stores, where order across continents does not matter.

Explain Spanner as a solution to an ordering problem, then show how each part solves one piece of it. Keep a small story, like the refund before its transfer, ready as your example.

1. 1

   State the problem

   Global transactions need one order that matches real time. Server clocks drift, so timestamps from different servers can flip the real order of events.

2. 2

   Show the storage

   Data is cut into splits by key range. Each split is a Paxos group with a leader, and a write commits on a majority vote.

3. 3

   Introduce TrueTime

   TT.now() returns a range that surely contains the real time. In the 2012 paper, GPS receivers and atomic clocks kept the clock error a few milliseconds either way.

4. 4

   Explain commit wait

   Pick the timestamp at the top of the range, then hide the write until that timestamp is surely past. Timestamp order now matches real order.

5. 5

   Cover writes across splits

   Two-phase commit runs over Paxos groups, so one crashed machine does not freeze the transaction. All participants commit at one timestamp.

6. 6

   Finish with reads and costs

   Snapshot reads use old row versions and safe time, with no locks. Then name the costs: slower writes, leader distance and a bigger bill.