Caching from First Principles: Strategies, Invalidation and Cache Stampedes

Picture an India vs Pakistan match. A wicket falls and a few lakh people refresh the score page in the same second. The score changes maybe once every 30 seconds, yet without a cache every refresh is a database query, and a Postgres primary that was happy at 2,000 queries per second is suddenly facing 200,000.

Caching is the obvious fix, and also where many incidents begin: stale prices, one user seeing another user's data, and the stampede where the cache expires and the database falls over anyway. This guide covers server-side caching; the browser side lives in our companion article on API caching with React Query, SWR and Apollo.

Caches work because fast storage is far faster than slow storage, and because real workloads read the same small set of data over and over. A cache is a smaller, faster copy of data whose source of truth lives somewhere slower. If either fact stops being true, the cache stops paying for itself.

From your app server: an in-process map lookup costs nanoseconds, a Redis call costs one datacenter round trip, and a database query costs a round trip plus planning, maybe a disk read and lock waits.

The second fact is locality and skew. Recently read data tends to be read again (temporal locality), and so does nearby data (spatial locality). Traffic is lopsided too: one live match gets more reads than a thousand archived ones, so a cache holding a tiny slice of data can serve most requests.

You can cache at every hop between the user and the disk. Layers closer to the user are faster and cheaper per request, but harder to invalidate, because you control them less.

• Browser cache: driven by Cache-Control headers. Free on a hit, but once you send max-age=3600 you cannot take it back.
• CDN edge: caches responses near the user. Great for assets and public pages; purges take an API call and some time.
• Reverse proxy: Nginx or Varnish can cache whole responses in front of your app.
• In-process cache: an LRU map inside each app server. Fastest server-side layer, but every pod has its own copy, so pods can briefly disagree.
• Distributed cache: Redis, Valkey or Memcached shared by all pods. One copy, one place to invalidate, one network hop.
• Database buffer pool: Postgres shared_buffers or the InnoDB buffer pool keep hot pages in RAM, for free.

For the score page: the CDN caches the public score JSON for a few seconds, each pod keeps a 1-second local copy, Redis holds the shared copy, and the database sees only misses. Personalised data, like a user's saved matches, must never sit in a shared cache; mark it Cache-Control: private.

Cache-aside, also called lazy loading, means the application checks the cache first and, on a miss, reads the database and writes the result into the cache with a TTL. The cache never talks to the database; your code does.

It is the default because its failure mode is gentle. If Redis restarts empty, reads get slower until the cache warms, but they stay correct, and memory only goes to data someone asked for. The costs: a miss pays three trips, and cached data can go stale.

Here is cache-aside in TypeScript with two production extras: single-flight (one in-flight load per key per process) and negative caching for rows that do not exist.

import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL ?? 'redis://localhost:6379');
const inFlight = new Map<string, Promise<unknown>>();
const MISSING = '__missing__';

// Spread expiries by +/-10% so keys written together do not expire together.
const withJitter = (ttl: number) => Math.round(ttl * (0.9 + Math.random() * 0.2));

export async function getOrLoad<T>(
  key: string,
  ttlSeconds: number,
  load: () => Promise<T | null>,
): Promise<T | null> {
  const cached = await redis.get(key).catch(() => null); // Redis down = treat as miss
  if (cached !== null) return cached === MISSING ? null : (JSON.parse(cached) as T);

  const pending = inFlight.get(key);
  if (pending) return pending as Promise<T | null>; // join the load already running

  const loadPromise = (async () => {
    try {
      const value = await load();
      const body = value === null ? MISSING : JSON.stringify(value);
      const ttl = value === null ? 30 : withJitter(ttlSeconds); // short TTL for "not found"
      await redis.set(key, body, 'EX', ttl).catch(() => undefined);
      return value;
    } finally {
      inFlight.delete(key);
    }
  })();

  inFlight.set(key, loadPromise);
  return loadPromise;
}

// Usage: const score = await getOrLoad('match:42:score', 5, () => db.getScore(42));

A Redis error is treated as a miss, so a cache outage means slower reads, not errors. The in-flight entry is removed in finally, so a failed load does not poison the key. The limit: single-flight is per process, so 40 pods can still send 40 loads for one key.

These patterns answer two questions: who loads the cache on a read, and when the database learns about a write. Read-through moves the loading into a cache library or proxy. Write-through writes cache and database together. Write-behind (write-back) writes the cache now and flushes to the database later.

My default: cache-aside, delete on write, and a TTL. Add write-through when a key needs read-after-write freshness. Use write-behind only for data you can afford to lose a few seconds of, like a like counter batched into the database. Never for money, seats or inventory: a Tatkal berth confirmed only in cache is a refund waiting to happen.

Pick the longest staleness a user can tolerate for that data, then go at or below it. The TTL is your safety net: even if an invalidation path has a bug, a stale value lives for one TTL at most, which is why every key needs one.

• Live score: 2 to 5 seconds. Nobody notices a 3-second delay.
• Product descriptions: minutes to hours, with a delete when the seller edits.
• Tatkal seat availability: cache the listing for a second or two to absorb the 10 AM refresh storm, but confirm the seat against the database inside the booking transaction.
• Session data: the session lifetime, with a delete on logout or password change.

Add jitter. If a deploy warms 50,000 keys with exactly 300 seconds of TTL, they all expire in the same second and the database sees a wall of misses. Plus or minus 10% turns the wall into a slope.

Delete the key after the database write commits, because a delete is idempotent and cannot put an older value on top of a newer one. Writing the new value into the cache opens a race between concurrent writers.

The update race, step by step

1. 1

   Writer A commits price 100

   A runs UPDATE price = 100 in the database.

2. 2

   Writer B commits price 200

   The database now correctly says 200.

3. 3

   B updates the cache first

   B is quick after commit and runs SET price 200.

4. 4

   A updates the cache last

   A had a GC pause, so its SET price 100 lands after B's.

5. 5

   The cache is wrong

   Database says 200, cache says 100, until the TTL expires.

With DEL instead of SET, order does not matter: the key ends up absent and the next read loads 200. Facebook's memcache paper uses this approach: a demand-filled look-aside cache where web servers delete keys after updating the database.

Delete has a smaller race too

1. 1

   Reader misses

   The key is absent, so the reader queries the database and gets the old price, 100.

2. 2

   Writer commits and deletes

   The writer sets 200 and deletes the key, which is already absent.

3. 3

   Reader writes the old value

   The slow reader runs SET price 100. Stale until TTL.

It is rare, but at scale rare happens daily. Fixes, cheapest first: short TTLs; a delayed second delete a second after the write; a version number stored with the value so older versions cannot overwrite newer ones; or leases. In the memcache paper, a miss hands the reader a lease token, a delete invalidates it, and the late SET is rejected.

Eviction is how a full cache makes room; expiry is what happens when a TTL runs out. LRU (least recently used) evicts whatever was touched longest ago. LFU (least frequently used) evicts whatever is touched least often, with decay.

Redis approximates both to save memory: it samples a few keys (maxmemory-samples, 5 by default), evicts the best candidate and keeps a pool of good candidates between rounds. LFU uses a small probabilistic counter per key. With skewed traffic, the approximation is close to the real thing.

A common trap: using Redis as a cache without setting maxmemory-policy. Memory fills, SET starts failing with OOM errors while reads keep working, and the volatile policies behave the same way if your keys have no TTL. Managed services may ship a different default, so check yours.

A hot key is one key getting enough traffic to saturate the node that owns it. In Redis Cluster every key lives on one slot on one primary, so adding shards does nothing for match:42:score during a final.

• Find them: redis-cli --hotkeys works only with an LFU policy; per-prefix client metrics work everywhere.
• Add a local layer: a 1-second in-process cache turns 100,000 Redis reads a second into one per pod per second.
• Replicate the key: write match:42:score#0 to #7 and read one at random. Invalidation must now delete all copies.
• Push instead of pull: fan out ball-by-ball updates over WebSockets so clients stop polling.

A cache stampede (also called a thundering herd or dog-piling) happens when a hot key expires and many concurrent requests miss at once, all rebuild the same value from the database, and overload it.

Think of Tatkal opening at 10 AM. If a popular train's availability is cached for 2 seconds, the key expires every 2 seconds while thousands of users refresh, and every in-flight request hits the database together. The rebuild slows under load, more requests pile up, and a 50 ms query becomes an outage.

Facebook's memcache paper reported that for keys prone to thundering herds, peak database load dropped from 17K queries per second to 1.3K once they added leases, with servers handing out a lease token only once every 10 seconds per key.

Fix 1: request coalescing (single-flight)

Concurrent misses for the same key in one process should share one load. Go ships this as golang.org/x/sync/singleflight; the helper above does it with a map of promises. Cheap and safe, but it only works inside one process.

Fix 2: a rebuild lock across processes

Let only the request that wins a short lock rebuild the key; the rest wait briefly and re-read, or serve the previous value. Give the lock an expiry so a crashed winner cannot block the key, and release it only if you still own it.

# Try to become the rebuilder. NX = only if absent, PX = auto-release after 3 s.
SET lock:match:42:score 7f3a91 NX PX 3000
# OK  -> you rebuild the value, then release the lock
# nil -> someone else is rebuilding; retry GET shortly or serve stale

# Release only if the token is still yours (one atomic Lua script)
EVAL "if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) else return 0 end" 1 lock:match:42:score 7f3a91

Fix 3: probabilistic early expiration (XFetch)

Each request rolls a die, and the closer the key is to expiry, the likelier that request is to refresh it early. Vattani, Chierichetti and Lowenstein showed in a 2015 VLDB paper that this scheme is optimal. In practice one request refreshes slightly early while everyone else keeps getting hits.

// Store expiresAt and delta (how long the last rebuild took, in ms) with the value.
// beta = 1 is the default; raise it to refresh earlier.
function shouldRefreshEarly(expiresAtMs: number, deltaMs: number, beta = 1): boolean {
  // Math.log(Math.random()) is negative, so this adds a random head start.
  return Date.now() - deltaMs * beta * Math.log(Math.random()) >= expiresAtMs;
}

Fix 4: stale-while-revalidate

Keep a soft deadline and a hard one. Between them, serve the stale value instantly and trigger one background refresh. HTTP standardised this in RFC 5861 as Cache-Control: max-age=600, stale-while-revalidate=30, with stale-if-error for serving old content while the origin returns 5xx. In Redis, store softExpiresAt inside the value and set the real TTL to the hard deadline.

Which to use? Single-flight always, since it is free. Stale-while-revalidate wherever a few seconds of staleness is fine. A lock or XFetch for expensive rebuilds like a leaderboard, or a scheduled refresh job so user requests never trigger the rebuild at all.

Cache penetration is when requests for keys that do not exist skip the cache every time, because there is nothing to cache. A bot walking /users/1 to /users/10000000 sends every request to the database while your overall hit ratio looks fine.

Negative caching is the first fix: cache "not found" with a short TTL, such as 30 seconds, so repeated lookups for the same missing key are absorbed. For attackers using a new random ID per request, also validate IDs before touching storage, rate limit by client, and consider a Bloom filter of existing IDs. A Bloom filter never gives a false "does not exist", so a filter miss can return 404 without a query.

• No TTL "because we invalidate explicitly". The first missed delete becomes permanent stale data.
• Updating the cache on write instead of deleting, or deleting before the database commit.
• Per-user responses behind a shared CDN or proxy cache, serving one person's data to another.
• One Redis for cache and durable data with allkeys-lru, so eviction quietly deletes the durable keys.
• Treating a Redis outage as fatal instead of degrading to slower reads behind a circuit breaker.
• Warming thousands of keys with identical TTLs, or caching multi-megabyte values that block a single-threaded Redis.

Next step: take one read-heavy endpoint you own and write down its hit ratio, TTL, invalidation path and what happens when Redis is down. Any "not sure" is your first fix.