What is a load balancer? Sharing requests across servers

SeatJaldi’s four app servers now remember nothing between taps, so any of them can answer anyone. Arjun points phones at them by putting all four addresses in DNS.

In a test, he switches off server 2 and deletes its address from DNS. Yet 1 in 4 test phones keep showing errors, long after the change.

“You removed it,” says Meera. “Why are people still going there?” Something at the door must pick a server for each request, and notice a dead one in seconds. What could that be?

A load balancer sits in front of several servers and passes each request to one healthy server.

• One door: every phone sends its request to the load balancer’s address.
• Healthy servers only: it keeps checking each server and skips any that stop answering.
• The door comes as two or more machines, because one broken door would stop everyone.

So why did phones keep going to server 2? Phones and resolvers keep a DNS answer until its TTL runs out, so the old address lived on. And DNS cannot tell whether a server is alive.

• A ticket site shows an error page, and a refresh loads it. One server was unwell, and a few requests reached it before the front door stopped sending it any.
• A friend says a site is crawling while it is quick for you. Their requests landed on a slow server.
• The padlock shows in your browser, yet the app’s own servers may never handle that encryption. Often the front door handles TLS.

By a rule. The simplest is the token machine’s “next counter in order”.

Taking turns is fair only if every clerk is equally quick. When one is slow, a better rule is “the counter with the shortest line”.

A third rule ties each user to one server. The load balancer turns the user id into a number with a hash function, and the number picks the server.

Some apps still keep a little on each server, and this rule brings the same user back to it, a sticky session. But adding or removing a server makes many user ids point at a different server.

The load balancer notices a dead server within seconds and stops sending it requests. If the door itself dies, a spare takes over.

It asks every server a small question every few seconds. Think of the bank manager glancing at each counter, and switching off the light of one that has stopped serving.

One load balancer alone would be a single point of failure, so real systems keep a spare, like the bank’s second token machine.

The answer: requests that reach server 2 before the third failed check get errors, for about 3 seconds. Then nobody is sent there.

A reverse proxy answers for the servers behind it; most load balancers are one.

The phone’s encrypted connection usually ends at this door. The request travels on to the servers behind the door, plain or encrypted again.

At Bikash’s bank, a guard checks his ID once, at the door, so no counter checks it again. A reverse proxy can check callers the same way.

Layer 4 load balancers pass each connection along without reading it, so they count connections, not requests, and usually leave the encryption to the servers. Layer 7 ones read the HTTP request and can send /seats calls to one server group and /payments to another.

Bigger apps split their work into services.

A big bank’s front desk sends you to loans or cash. An app’s version is an API gateway. Keep the door thin: the more it does, the more can break.

• Weighted round robin: a server with weight 2 gets two turns for every one.
• Least request with two random choices: pick 2 servers at random and use the less busy one. It is nearly as good as checking every server.
• Ring hash: servers and user ids sit on a circle, and each user goes to the next server clockwise. Adding or removing one of N servers moves only about 1 in N users.

An active health check calls a small /health page on a timer. A passive one marks a server down when real requests fail.

Assume a check every second and 3 failures to mark a server down: about 3 seconds. At 800 requests a second over four servers (4 × 200), a dead one gets 200 × 3 = 600 failed requests. With a 30-second check and 2 failures, it gets about 200 × 60 = 12,000.

What just happened? Detection time times traffic is the damage. At the bank, that is three seconds of people walking up to a counter whose light is still on. Passive checks shrink it, and the door can retry an idempotent request on another server.

Try it yourself

1. 1

   Set up

   Servers A and B serve one request at a time. Requests 1 to 6 arrive at t = 1 to 6 seconds.

2. 2

   Sizes

   Request 1 takes 6 seconds, the rest 1 second each. A request ending as another arrives counts as finished.

3. 3

   Round robin

   Odd requests go to A, even to B. Write each time (finish minus arrival) and the average.

4. 4

   Least request

   Each request goes to the server with fewer in progress; ties go to A.

5. 5

   Check your work

   Round robin totals 18 seconds, an average of 3.0. Least request totals 11, about 1.8. Round robin piled work behind the heavy request.

Back in Kothrud, Arjun puts a load balancer pair in front of the four servers, and DNS points at the door. Who decides which server gets each request? The door, by a rule.

The noon test at about 800 booking requests a second passes. And if a server dies? A health check stops its errors within seconds. But three servers carry only about 600 a second, so Meera wants a spare server for sale day.

Then the database’s processor climbs to 95%. Those 800 a second are bookings, but every fan’s phone also keeps refreshing the seat map. So all four counters keep fetching the same file from one cabinet, about 10,000 times a second. Why ask the database every time?