Clock Synchronization Explained: Why Server Timestamps Lie and What to Do

Each server stamps events with its own clock, and even with clock synchronization no two clocks agree exactly. When code sorts events by those stamps, it can put them in the wrong order and pick the wrong winner.

Picture a common setup. An online spice auction in Kochi sells cardamom to about 300 traders every morning, in 140 lots. The setup is made up, but the problem is real.

Each lot is one batch of cardamom, such as 600 kg of green pods. A lot closes at a fixed time, and the highest bid stamped before the close wins. Equal bids go to whoever bid first.

Bids arrive at two bid servers, A and B, behind a load balancer (a box that spreads requests across servers). Each bid server stamps a bid with its own clock time when the bid arrives.

One Tuesday, lot 37 closes at 11:00:00. Server B's time service had stopped after a maintenance move a week earlier, and its clock now runs 3 seconds slow. Nobody noticed.

• A trader bid Rs 2,400 a kg through server A at 10:59:57. Another bid the same amount through server B two seconds later, but B stamped it 10:59:56, so it looked first.
• A third trader bid Rs 2,410 a kg through server B two seconds after the close. B stamped it 10:59:59, so the system accepted it, and it won the lot.

Both stamps were wrong, and the code trusted them completely. This guide explains why clocks disagree, how they are kept in line, and how the auction team stopped clocks from picking winners.

Clock synchronization in distributed systems is the ongoing work of keeping every machine's clock close to one shared reference time, usually UTC. A distributed system is a group of computers working together over a network.

UTC, Coordinated Universal Time, is the world's reference time. Indian time is UTC plus 5 hours 30 minutes. Servers usually keep UTC inside and convert to local time only for display.

Picture a watch repair shop under the old clock tower of a bazaar. Every wristwatch on the bench gains or loses a little each week. The repairer looks up at the tower clock and sets each watch to match it.

Your servers are those wristwatches, and a time server is the tower clock. Clock synchronization is the repairer's job, done by software every few minutes, for as long as the servers run.

Where the picture breaks: the repairer sees the tower instantly. A server must ask over a network, and the answer arrives late by an amount it cannot measure exactly. That uneven delay limits how exact the setting can be.

Four terms you will see everywhere

Clock drift. Drift is how fast a clock wanders away from true time. A clock that gains 1 second a day has a drift of about 12 parts per million, written ppm.

Clock skew. Skew is the gap between two clocks at one moment. At 11:00 on that Tuesday, the skew between server A and server B was 3 seconds.

Offset. Offset is the gap between one clock and the reference time. Server B's offset was 3 seconds: you had to add 3 seconds to its reading to get UTC.

Wall clock and monotonic clock. The wall clock tells the time of day, and software can change it. A monotonic clock only counts forward from some start point, so it suits measuring how long things take.

Clocks drift for a physical reason. Most servers count time with a small quartz crystal that vibrates at a set rate. Heat, age and tiny manufacturing differences make each crystal run slightly fast or slow.

The NTP standard, RFC 5905, budgets for a clock rate error of up to 15 ppm. At that rate, a clock with nothing correcting it would gain or lose about 1.3 seconds a day, or 9 seconds a week.

Many people picture synchronization as making clocks equal. A better picture is that it shrinks the gap to a size you can measure. The gap never reaches zero, and it starts growing again the moment a correction ends.

NTP, the Network Time Protocol, asks a time server for the time and allows for how long the messages spent on the network. It repeats this every few minutes and adjusts the local clock gently.

The hard part is the trip. If the answer takes 20 ms to arrive, the time inside it is already 20 ms old. NTP measures the round trip with four timestamps and assumes each direction took half.

Follow the four boxes in order. Notice which clock writes each timestamp: T1 and T4 come from server B's own clock, while T2 and T3 come from the time server's clock.

The NTP standard gives two small formulas. Offset is how far server B's clock is from the time server's clock. Delay is how long the messages spent on the network, there and back.

• Offset equals ((T2 minus T1) plus (T3 minus T4)) divided by 2.
• Delay equals (T4 minus T1) minus (T3 minus T2).

Here is that check of server B as a tiny Python function. The four times are seconds after 10:00:00, each read from the clock of the machine that wrote it.

def ntp_offset_and_delay(t1, t2, t3, t4):
    """t1 and t4 come from our clock, t2 and t3 from the time server."""
    offset = ((t2 - t1) + (t3 - t4)) / 2
    delay = (t4 - t1) - (t3 - t2)
    return offset, delay

# Bid server B asks a time server (seconds after 10:00:00 on each clock)
offset, delay = ntp_offset_and_delay(0.000, 3.020, 3.021, 0.041)
print(f"offset {offset:.3f} s, delay {delay:.3f} s")  # offset 3.000 s, delay 0.040 s
print(f"possible error: up to {delay / 2:.3f} s")      # possible error: up to 0.020 s

The offset of 3.000 seconds says server B is 3 seconds behind. The delay formula removes the 1 ms the time server spent between T2 and T3. Run it with Python 3 and you get exactly the two lines in the comments.

The last line matters most: the offset can be wrong by up to half the delay, here 20 ms. NTP assumes the request and the reply took equal time, and it cannot check that.

Suppose the request really took 5 ms and the reply 35 ms. We ran that case through the same function: it reports 2.985 seconds instead of 3.000, an error of 15 ms. Short, even network paths give better time.

Who tells the time server the time?

Time servers form layers called strata. A stratum 1 server reads time directly from a reference clock, such as a GPS receiver. Stratum 2 servers sync from stratum 1 servers, and so on down.

In the NTP standard, stratum 16 means a server is not synchronized at all. Each layer adds a little error, so fewer layers usually means better time.

Here is how the auction gets its time after the fix. Read it from the top: time flows down from the reference clock to the bid servers, and both bid servers report to a monitor.

Check your own server's offset

If your Linux server runs chrony, an open-source NTP program, run chronyc tracking. The System time line is the current offset between NTP time and your system clock.

The chrony 4.9 manual shows a healthy example: System time : 0.000006523 seconds slow of NTP time. That is about 6.5 microseconds. Its Leap status line can read Normal, Insert second, Delete second or Not synchronised.

Then run chronyc sources. A star marks the source chrony has selected. An x marks a falseticker, a source whose time disagrees with the majority of the others.

NTP is accurate to a few hundred microseconds for typical clients on fast local networks, says the standard. Across the internet it is looser, because the error can reach half of a longer, more uneven round trip.

Cloud providers publish their own figures. In November 2023, AWS said supported EC2 instances, its virtual servers, typically see a clock error bound under 100 microseconds over its NTP service. With its PTP (Precision Time Protocol) hardware clock, a time source built into the server, it said under 40 microseconds.

Treat those as the vendor's typical numbers, not a promise. For the auction, two healthy bid servers in one data centre might disagree by well under a millisecond. A trader's phone could be off by much more.

Slewing versus stepping

When NTP finds an error, it can fix it two ways. Slewing means running the clock a tiny bit faster or slower until it catches up. Stepping means jumping the clock straight to the new time.

The standard slews small errors and steps only when the error passes a step threshold of 125 ms. Above its panic threshold of 1,000 seconds, the program should exit with a message to the system log, so a person sets the clock.

Real programs set their own rules. chrony 4.9 normally slews, and its configuration file allows steps through the makestep directive. The manual's example, makestep 0.1 3, steps errors over 0.1 seconds during the first three updates only.

chrony's default slew limit is one twelfth, written 83333.333 ppm. So slewing away a 1 second error takes at least 12 seconds. Back at the watch shop, that is the repairer regulating a watch instead of twisting its hands.

Stepping backward is the dangerous case. If the wall clock jumps back 3 seconds, any code that subtracts two wall clock readings can get a negative duration.

Leap seconds and smearing

A leap second is an extra second added to UTC now and then, to keep it in step with the Earth's uneven spin. Some systems show it as 23:59:60, and others repeat 23:59:59.

Some large operators smear the leap second instead. Google documents a linear smear over 24 hours, noon to noon UTC, and says its public NTP and AWS follow the same practice. Each second gets about 11.6 microseconds longer, and clocks never jump.

The price is that smeared clocks sit up to about half a second away from official UTC during the smear. AWS smears on its NTP service but not on its PTP hardware clock. It advises against mixing smeared and non-smeared sources in one time client during a leap second.

Other ways clocks go wrong

• The NTP service stops, as on server B, and the clock drifts with no alarm.
• A firewall blocks NTP's network port, UDP 123, so every sync request quietly fails.
• Someone sets the clock by hand to test something, and forgets to undo it.
• A program pauses, for example during garbage collection (the runtime freezing to clean memory), so a correct reading is used late.

You should measure durations with a monotonic clock because it only moves forward, while the wall clock can jump backward when corrected. Subtracting two wall clock readings can then give a negative time.

This has hurt real companies. A leap second was added at the end of 31 December 2016 UTC. Cloudflare's DNS software measured response times with Go's time.Now(), which did not guarantee monotonic readings then.

DNS is the internet's address book, which turns names into server addresses. When time went back a second, some measured durations came out negative. One reached a random number function that panics on negative input.

Cloudflare's report says that at peak, about 0.2% of its DNS queries were affected. The fix stopped negative values reaching the server selection code. Go 1.9 later made time values carry a monotonic reading, so subtracting them became safe.

Python makes the choice explicit. Its documentation warns that time.time() can return a lower value if the system clock is set back. By contrast, time.monotonic() is not affected by system clock updates, and only the difference between two readings means anything.

You can check which clock Python uses on your machine. This is what we got on Python 3.13.16 on Linux:

$ python3 -c "import time; print(time.get_clock_info('monotonic'))"
namespace(implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, adjustable=False, resolution=1e-09)

Monotonic does not mean untouched. On Linux, CLOCK_MONOTONIC ignores jumps in system time, but its rate follows NTP's frequency adjustments, and it stops counting while the machine is suspended. Python's adjustable=False refers to jumps, not to those gentle rate changes.

So here is which clock to use for which job:

• Time of day for logs, receipts and display: time.time(), kept in UTC.
• How long something took: time.monotonic().
• Timeouts, retries and countdowns: time.monotonic().
• The order of events across machines: neither clock. That needs one of the designs later in this guide.

TrueTime is the clock service behind Spanner, Google's globally distributed database, and it returns a time range instead of a single time. The 2012 Spanner paper says the real time is guaranteed to lie inside that range.

The paper calls the main function TT.now(). It returns an interval with an earliest and a latest time. Half the width of that range is called epsilon: the clock's admitted uncertainty.

Think of a very honest repairer at the watch shop. Asked the time, he does not say 4:10. He says it is between 4:09 and 4:11, and he is certain of that.

To keep epsilon small, Spanner runs time masters in every data centre. Most have GPS receivers. The rest have atomic clocks, because the two kinds of reference fail in different ways.

The paper reports that epsilon swings from about 1 to 7 ms over each poll interval, so it is about 4 ms most of the time. Machines poll the masters every 30 seconds.

Those numbers come from a simple budget. Between polls, a machine assumes its clock drifts by 200 microseconds per second, which adds up to 6 ms over 30 seconds. The other 1 ms is the delay to reach the time masters.

The paper also admits excursions. A time master outage can raise epsilon across a data centre, and overloaded machines or network links can cause local spikes.

Commit wait: waiting out the doubt

Spanner gives each transaction (a group of changes saved together) a commit timestamp, no earlier than TT.now().latest. Before anyone may see the result, the leader, the machine in charge of that group of data, waits until TrueTime says that timestamp has definitely passed.

The paper calls this commit wait. Pause and predict: if epsilon is 4 ms, roughly how long must each commit wait?

The paper says the expected wait is at least twice epsilon, so about 8 ms here. In return, if transaction B starts after transaction A's commit, B gets a larger timestamp than A, even across continents.

We checked the rule in a toy model of our own, not Spanner's code. Two machines got random clock errors inside epsilon, and transaction B always started after A's commit was visible.

The 16.7% comes from our made-up workload, and it does not change with epsilon because everything in the model scales together. Only the zero is guaranteed by the rule. The wait grows with epsilon, which is why Spanner works to keep epsilon small.

Here is the real lesson. Spanner does not trust clocks more than other systems do. It trusts them less, measures exactly how much less, and pays for that doubt in waiting time.

You do not need Google's hardware to get an error bound. On EC2, AWS's open-source ClockBound daemon measures an instance's clock error bound. It lets you check whether a given timestamp is in the past or future.

Timestamps give a trustworthy order in two cases. Either the events are further apart than the clock error, or every stamp comes from one source that never steps back, such as a monotonic clock or a counter. Inside the error window, the order is a guess.

Healthy bid servers in one data centre might agree within a millisecond. Two bids 2 seconds apart are then safe to order by timestamp. Two bids 0.3 ms apart are not.

Use this decision path when a design needs events in order. Start at the top question and follow the answers to the cheapest tool that is still correct.

Logical clocks need a short word here. A Lamport clock is a counter that each machine bumps on every event and sends along with every message. On receiving a message, a machine moves its counter past the number it received. It reads no real clock at all.

If event A could have affected event B, A gets the smaller Lamport number. The reverse does not hold: a smaller number alone does not prove A came first.

Vector clocks extend the idea, so you can also tell when two events happened independently, with neither affecting the other. Both ignore the time of day completely.

The team made one process per lot, a lot sequencer, the only place that decides bid order and closing. Clocks still stamp bids for people to read, but they no longer pick winners.

A sequencer is a single process that gives each event the next number in a line: 1, 2, 3. Because only one process numbers the bids for lot 37, there is no second clock to disagree with.

Here is the core of it, with a short run at the bottom. In production, each lot gets one LotSequencer object on one machine, open for 300 seconds and timed with that machine's monotonic clock.

import itertools
import time

class LotSequencer:
    """One per lot. The only place that decides bid order and the close."""

    def __init__(self, lot_id, open_for_s):
        self.lot_id = lot_id
        self.seq = itertools.count(1)
        self.closes_at = time.monotonic() + open_for_s  # immune to clock steps
        self.closed = False

    def accept(self, trader, amount_rs):
        if self.closed or time.monotonic() >= self.closes_at:
            self.closed = True
            return None                                  # late: refuse the bid
        return (next(self.seq), trader, amount_rs)       # place in line breaks ties


lot = LotSequencer("lot-37", open_for_s=0.5)  # half a second, to try it quickly
bids = [lot.accept("trader-1", 2400), lot.accept("trader-2", 2400)]
time.sleep(0.6)
print(lot.accept("trader-3", 2410))                       # None: too late
best = max(bids, key=lambda b: (b[2], -b[0]))            # highest amount, then lowest number
print(bids, "winner:", best[1])

Run it and you get None, then [(1, 'trader-1', 2400), (2, 'trader-2', 2400)] winner: trader-1. We ran it five times with the same output.

Two lines carry the design. The closing check uses time.monotonic(), so a wall clock step cannot reopen or shorten the lot. next(self.seq) gives each accepted bid its place in line, and equal bids go to the lower number.

The countdown is only as right as its start. Open the lot when the machine's NTP-synced wall clock reads the agreed time, then count with the monotonic clock.

Here is the new bid path. Follow a bid from the trader's phone to the database, and notice that the bid servers no longer decide anything.

What if the sequencer's machine dies?

A standby takes over through leader election, a way for machines to agree which one is in charge. A monotonic deadline means nothing on the standby, so it recomputes the close from the stored UTC close time.

Many leader-election setups use leases, permissions that expire after a set time. The 1989 paper that introduced leases says they need clocks with a known, bounded drift. A server clock that runs too fast can allow a write before the old holder's lease expires.

So the team keeps lease timeouts generous compared with the clock error its monitor allows. Here is the order in which it rolled the changes out:

1. 1

   Fix server B

   Restart its NTP service, confirm the offset falls under 1 ms, and find out why the maintenance move stopped it.

2. 2

   Add offset alerts

   Every server reports its measured offset each minute. An alert fires if any server passes 50 ms.

3. 3

   Move ordering to sequencers

   Route every bid for a lot to that lot's sequencer, which numbers the bids and closes the lot.

4. 4

   Switch durations to monotonic time

   Countdowns, timeouts and response time numbers now use time.monotonic(), never time.time().

5. 5

   State the closing rule

   Update the trader app: a bid counts when the auction system receives it, not when the phone sends it.

The team settles the Tuesday dispute by hand, using the load balancer's arrival log, which ran on a healthy clock. The first trader gets lot 37. From the next morning, the sequencer makes such calls on its own.

Explain it in layers: clocks drift, NTP narrows the gap, a gap always remains, and so any ordering that matters must not rest on timestamps from different machines.

1. 1

   Say why clocks disagree

   Quartz drift, budgeted at up to 15 ppm or about 1.3 seconds a day, plus network delay that blurs every attempt to correct it.

2. 2

   Explain NTP in one breath

   Four timestamps give offset and delay. The client uses several servers, slews small errors and steps large ones.

3. 3

   Give the error budget

   A few hundred microseconds on a fast local network, looser across the internet, and far worse when sync silently stops.

4. 4

   Separate the two clocks

   Wall time for display and logs, monotonic time for durations, timeouts and countdowns.

5. 5

   Pick an ordering tool

   A sequencer when one place sees every event, logical clocks for cause and effect, TrueTime-style waiting for global transactions.

6. 6

   Name the monitoring

   Alert on the measured offset, watch for clock steps, and decide how every machine handles a leap second.