How does caching work? Keeping a copy close by

The load balancer works, and the noon test’s 800 bookings a second spread across SeatJaldi’s four servers. Yet the database’s CPU, the chip that does its thinking, is 95% busy.

Arjun checks the dashboard. One question keeps the database busy: “give me the seat map for match 42.” Every fan’s phone keeps refreshing it: about 10,000 times a second. Each time, the answer is worked out from scratch.

Meera frowns. “The picture barely changes from one tap to the next. Why are we doing the same work 10,000 times?” Arjun suggests keeping a copy, and Meera asks the harder question. “How old can that copy be before someone taps a seat that is already gone?”

A cache keeps a copy of data in a faster, closer place, so the next request can skip the slow source. You met caches with DNS answers. In one line: you trade freshness for speed.

• A copy close by: kept nearer than the source.
• Fast: reaching it is far quicker than asking the source.
• Maybe old: the source can change after the copy was made.

A cache hit means the copy was there, so the answer came back fast. Kavita finds masala in the dabba.

A cache miss means the copy was not there, so the source answers and the answer is saved. That is the walk to the store room.

Out of every hundred times Kavita reaches for the dabba, how often does she find masala? That share is the hit rate.

• A website is slow the first time and instant the second. The second visit used the stored copy.
• A friend still sees your old profile photo. Their phone kept the old copy until its time limit ran out.
• A cricket score sits still for a few seconds, then jumps. The page serves a copy on a short timer.

Copies can live in four places, nearest first. They are the user’s phone or browser, copies kept in many cities near users, memory beside the app servers, and the database’s own memory.

Kavita tries the platform dabba first, then the bigger family dabba in the next room, then the store room. Each dabba has its own lid date. A dabba can hold old masala while the store room gets a fresh pack, and a copy can too.

The database’s own memory is different: it is updated on every write, so it never goes stale.

A cache hurts when the hit rate is low, when it is empty, or when it holds data that must be exact. Call the hit rate h. Then average = h x hit + (1 - h) x (hit + database), the same sum as step 4 of the walk-through.

These inputs are assumptions, not measurements. A cache hit costs 0.5 ms, about one round trip inside a data centre. A database read costs 5 ms. A miss pays both, 5.5 ms.

Often the bigger win is protecting the database, not speeding up users. At 95% hits, 10,000 reads a second send only 10,000 x 0.05 = 500 to the database.

A cold start is a new kitchen with an empty dabba. After a restart the hit rate is 0, so for a moment all 10,000 reads a second reach the database: 20 times its usual 500.

• Pick the TTL by asking how stale this may be before someone is hurt. Seat map: 2 seconds. Match schedule: hours.
• You would never keep milk in the masala dabba: it turns in hours, and you cannot tell by looking. Never cache payment status either; it must be right every time.
• Name each copy by everything that makes it different: seatmap:match-42 keeps match 43 from getting match 42’s map. Personal data needs the person in the name too.
• Ages stack: a 2-second phone copy of a server copy already 1.9 seconds old can be almost 4 seconds old. HTTP has an Age header, a number saying how old a copy is, so every cache can add up the ages.
• A tiny dabba refilled all day is a low hit rate. If requests rarely repeat, a cache only adds a step.

Try it yourself

1. 1

   Write the formula

   average = h x 0.5 + (1 - h) x (0.5 + 5).

2. 2

   Put h = 0.8

   Work out the hit part and the miss part, then add them.

3. 3

   Work out the database load

   Multiply 1 - h by 10,000 reads a second.

4. 4

   Check your work

   You got it right with an average of 1.5 ms and 2,000 database reads a second, four times the 500 at 95%.

5. 5

   Extra: the break-even

   Solve h x 0.5 + (1 - h) x 5.5 = 5 to get h = 0.1. Below a 10% hit rate, the cache is slower than none.

What just happened? At 80% hits the average only doubled, from 0.75 ms to 1.5 ms, but the database’s work grew four times.

Back at the dashboard, the database CPU line drops. Now 95 of every 100 seat-map requests never reach it: 20 times less work.

Meera asks again how old the copy may be. The TTL answers it, like the date on the dabba’s lid. The copy is never older than 2 seconds, and booking still checks the database. At worst, someone sees “that seat was just taken”. You pay for speed with freshness; the TTL is the price tag you choose.

Then Arjun looks at the four servers on his diagram. Where does this cache live, if there are four servers? Everyone says “use Redis”.