Load Balancers Explained: L4 vs L7, Global vs Local

One broken server fails a share of all users, because nothing in front of the servers notices it and steers traffic away. Spotting and skipping that server is the first job of a load balancer.

Picture a common setup. A ticketing site sells concert and comedy tickets from four app servers, called app-1 to app-4. Its domain name points straight at all four addresses through DNS, the internet's phone book that turns names into addresses. The setup is made up, but the failure is real.

Tickets for a popular comedian's tour open at noon on a Friday. About 60,000 people arrive in the first ten minutes. At 12:04, app-3 runs out of memory and stops answering, but its address stays in DNS.

• About one buyer in four keeps reaching app-3 and sees a spinning page, then an error.
• Phones that saved app-3's address keep trying it for minutes, even after the on-call engineer removes it from DNS.
• Buyers who refresh onto another server lose the seats they held, because seat holds live in each server's own memory.
• app-1 sits at 30 percent CPU while app-2 runs at 95 percent, because DNS split users unevenly.

The team's fix has four parts, and they match the next sections. Put a load balancer in front, pick a sensible algorithm, add health checks, and move seat holds out of server memory.

A load balancer is a server or managed service that receives every request for your site and passes each one to one of several backend servers. It spreads the work and skips servers that are down.

Users now talk to one address: the load balancer's. Behind it sits the pool, the group of backend servers doing the real work; AWS calls each one a target. Users never see app-3's own address, so a broken app-3 can be hidden quickly.

Think of a busy petrol pump on the Pune to Mumbai highway. An attendant at the entrance waves each vehicle to a free nozzle. Drivers do not pick a nozzle themselves, so no queue forms at one while others sit idle.

• The attendant is the load balancer, and the nozzles are the servers in the pool.
• Choosing which nozzle gets the next car is the algorithm.
• Spotting a nozzle that shows an error, and skipping it, is the health check.

Where the picture breaks: the attendant sees every nozzle directly. A load balancer only learns that a server is sick from probes and failed requests, so it always finds out a little late.

A load balancer gives the site three things.

• Spreading load. Work is shared, so 60,000 buyers become about 15,000 per server instead of a pile on one.
• Hiding failures. A dead server is skipped once health checks catch it, from about 15 seconds to a minute depending on settings. Nobody waits for phones to forget an old address.
• Easy changes. You can add, remove or update servers behind the single address while users notice nothing.

Here is the site after the team's first change. Notice that the shared session store sits beside the servers, not inside any one of them.

An L4 load balancer routes each connection using only network details, like IP addresses and ports, without reading the request. An L7 load balancer reads each HTTP request and can route on its path, headers or cookies.

The numbers come from the OSI model, a seven-layer way of describing networks. Layer 4 is the transport layer, where TCP and UDP carry data between two machines. Layer 7 is the application layer, where HTTP, the language of browsers and web servers, lives.

L4: fast, and blind to the request

An L4 balancer sees a new TCP connection arrive and picks a server for it. Every packet, a small chunk of data, in that connection then goes to the same server. It never opens the envelope, so it cannot tell a seat booking from an image download.

AWS documents this for its Network Load Balancer. It picks a target with a flow hash. A hash is a function that turns an input into a number, always the same number for the same input.

Here the input is the protocol, both addresses and ports, and the TCP sequence number, a byte counter that starts at a different value for each connection. So each connection stays with one target for its whole life, but two connections from the same phone can land on different targets.

Because so little work happens per packet, L4 is fast and can carry TCP traffic and, in many products, UDP traffic, not just web pages. L4 balancers come in two builds. Some forward packets without ending the connection: Google's Maglev paper reports one machine filling a 10 Gbps (10 gigabits a second) link with small packets.

Others, like HAProxy in TCP mode, end the client's connection and open their own to the server. We checked on HAProxy 2.8.16: the server saw a different source port from the client's, yet received the bytes unchanged. Neither build reads the request.

L7: more work, but it reads the request

An L7 balancer ends the user's connection, reads the HTTP request, then sends it to a backend over its own connection. That is why it is also called a reverse proxy: a server that receives requests on behalf of other servers.

Reading the request lets it make choices L4 cannot. It can send seat booking requests to booking servers and search requests to search servers. It can read cookies, add headers, retry a failed request elsewhere, and handle HTTPS, the encrypted form of HTTP, in one place.

AWS's Application Load Balancer works this way. It checks listener rules, the routing rules you write, in priority order from the lowest number. The first match names a target group, which is AWS's word for a pool, and that group's algorithm picks a server.

It also adds an X-Forwarded-For header, an extra labelled line on the request, so the backend learns the buyer's real address. Without it, every request would seem to come from the load balancer.

The team uses both, which is a common pattern. A fast L4 layer takes every connection from the internet, and an L7 pair behind it reads each request and picks a pool. Follow one request below and watch where the path starts to matter.

Use round robin when servers are equal and requests take similar time, and least connections when request times vary a lot. Use a hash only when the same user or key must keep landing on the same server.

The algorithm is the rule that picks a server for each new request or connection.

Round robin and weighted round robin

Round robin hands out requests in turn: app-1, app-2, app-3, app-4, then back to app-1. It is the default in nginx, a widely used open-source web server and proxy, and in AWS's Application Load Balancer. It is fair when every request costs about the same.

Weighted round robin gives bigger servers more turns. The nginx docs give one server a weight of 3 and two others a weight of 1, so every five requests split three, one and one. On nginx 1.24.0 we saw app-1, app-2, app-1, app-3, app-1, repeating.

Least connections

Least connections sends each new request to the server with the fewest requests in progress right now. AWS's version is called least outstanding requests. AWS says it is commonly used when requests vary in complexity or targets vary in processing capability.

That fits the ticketing site. Loading an event page takes about 20 milliseconds, but holding seats takes about 400, because it locks rows in the database. Round robin counts both as one turn, so a server that gets several seat holds in a row falls behind.

This is the petrol pump attendant at his best. He does not wave vehicles to nozzles in strict turn. He watches which nozzle is actually free, because a bike fills in one minute and a loaded truck takes five.

Power of two choices

Checking every server's count on every request gets costly with hundreds of servers. So some proxies pick two servers at random and send the request to the less busy one. This trick is called the power of two choices.

Envoy, an open-source proxy, does this in its least request balancer when servers have equal weights. nginx offers it as the random two method. Envoy's docs cite research showing it is nearly as good as checking every server.

Hashing

A hash method turns something about the request, like the user's IP address or a cookie, into a number that picks the server. The same input lands on the same server as long as the pool does not change. nginx's ip_hash method works this way.

Plain hashing has a catch. It takes the hash modulo the number of servers, which means the remainder after dividing, so changing that number moves most users. Consistent hashing places servers and users on a ring, so only a small share moves.

We tested both with 100,000 made-up user IDs. When app-3 left, plain hashing moved 75 percent of users to a new server. A consistent-hash ring moved 24.6 percent, which is just app-3's own users.

Here is the heart of a least-busy picker in Python. Each server object has an in_pool flag, set by health checks, and an active count of requests in progress.

import random

def pick_server(servers):
    pool = [s for s in servers if s.in_pool]
    if not pool:                     # nothing healthy: fail open
        pool = servers
    if len(pool) == 1:
        return pool[0]
    a, b = random.sample(pool, 2)    # power of two choices
    return a if a.active <= b.active else b

What just happened? random.sample draws two different servers, and the last line keeps the one with fewer active requests. The fallback near the top sends traffic to every server when none look healthy, which the next section explains.

We ran it 30,000 times with app-3 out of the pool and active counts of 2, 9 and 5. It chose app-1 about two times in three and app-4 one time in three. It never chose the busiest server, app-2, because that one loses every pair it is drawn into.

A health check is a small test request the load balancer sends to each server on a timer. If a server fails a set number of checks in a row, it leaves the pool. When it passes enough in a row, it comes back.

These are active health checks, because the load balancer goes looking. AWS's Application Load Balancer sends an HTTP GET, a request that only reads data, every 30 seconds by default for instance and IP targets. It waits 5 seconds for a reply and expects status code 200, which means OK.

Two thresholds stop flapping, which means servers bouncing in and out of the pool on every small hiccup. By default, AWS marks a target unhealthy after 2 failures in a row. It needs 5 passes in a row to mark it healthy again.

Passive health checks watch real traffic instead. nginx counts failed attempts to reach a server within a window called fail_timeout, 10 seconds by default. With the default max_fails of 1, a single failure takes the server out for that window.

Follow one probe from the timer to its end. The team's checker probes every 10 seconds, and a server never leaves or returns on a single result.

The team's checker keeps two counters per server, following the diagram.

FALL = 2   # failures in a row before a server leaves the pool
RISE = 3   # passes in a row before it comes back

def record_probe(server, passed):
    if passed:
        server.passes += 1
        server.fails = 0
        if not server.in_pool and server.passes >= RISE:
            server.in_pool = True
    else:
        server.fails += 1
        server.passes = 0
        if server.in_pool and server.fails >= FALL:
            server.in_pool = False

The key detail is that each result resets the opposite counter. So fails always means failures in a row, not failures ever. If a probe starts every 10 seconds with a 2-second timeout, a frozen server leaves the pool 12 to 22 seconds after it freezes.

The spread depends on where the probe timer was when the server froze. On HAProxy with the same settings we measured about 14 to 24 seconds; the engineer section explains the gap.

What should a health check test?

A good health endpoint quickly checks what the server needs to sell a ticket. The team's endpoint, a small page called /healthz, confirms the app can reach the session store and the database within 200 milliseconds.

Do not make it too deep, though. Suppose every server's check also called the payment provider, and that provider slowed down. Every server would fail at once, and the load balancer would empty the pool over a problem it cannot fix.

Draining and slow start

Health checks handle surprise failures. For planned removals, like a deploy, use draining: the balancer stops sending new requests to a server but lets requests in progress finish. AWS calls this the deregistration delay and waits up to 300 seconds by default, finishing early once a server has no requests or connections left.

The reverse is slow start. By default, AWS gives a new target its full share as soon as it passes one health check. Slow start mode instead raises its share linearly, so caches and database connections can warm up. In nginx, slow_start is part of the paid NGINX Plus subscription.

Sticky sessions, also called session affinity, make the load balancer send all of one user's requests to the same server. Avoid them when you can, by keeping session data in a shared store instead.

The ticketing site leaned on stickiness by accident. Each server kept seat holds in its own memory, so a buyer moved to another server lost the seats. Stickiness hid that problem until app-3 died.

There are two common ways to stick. You can hash the user's IP address, as nginx's ip_hash does. Or you can use a cookie, a small piece of data the browser sends back with every request, that names the server.

AWS's Application Load Balancer can make its own cookie, called AWSALB, with a stickiness duration you set from 1 second to 7 days. If that cookie points to a target that is unhealthy or deregistered, the balancer picks a new target and updates the cookie.

• Uneven load. A server that picked up many long sessions stays busy, while a newly added server gets little traffic until old cookies expire.
• Lost state. When the sticky server dies, its memory dies with it, so the user loses the session anyway.
• Shared addresses. nginx's ip_hash uses only the first three parts of an IPv4 address, the common four-number form such as 127.0.0.1. In our test, clients 127.0.0.1, 127.0.0.2 and 127.0.0.200 all landed on app-1.

So a whole office or mobile network sharing one address range can pile onto one server. The team's fix was to move seat holds into a shared session store, a fast in-memory database every server reads. Now any server can handle any request.

That makes the app servers stateless, meaning they keep nothing a user needs between requests, and stickiness can be switched off.

Global load balancing picks which region or data centre a user goes to. Local load balancing picks which server inside that region handles the request.

The team's next goal is a second region, with servers in Mumbai and Hyderabad, so one region's outage cannot stop a sale. A local load balancer cannot help with that, because it lives inside one region.

Global balancing is often done with DNS. When a phone asks for the site's address, the DNS service answers by rules instead of giving one fixed answer. Amazon Route 53, for example, offers latency, geolocation, weighted and failover routing policies, among others.

• Latency routing sends users to the region that gives them the best latency, which is how long a request takes to get there and back.
• Geolocation routing sends users by where they are, such as every user in one country to one region.
• Weighted routing sends set shares of users to each place, such as 90 percent to one region and 10 percent to another.
• Failover routing sets up active-passive failover: users go to a backup region when the main one is unhealthy.

DNS has the same weakness that hurt the ticketing site. Resolvers, the DNS servers that look up names for your phone, may cache each answer for up to its TTL, the record's time to live. So a change reaches users only as those copies expire.

Here is the team's two-region plan. Look for the two layers: one global decision at the top, then a normal local setup inside each region.

You run more than one load balancer and give them a way to share or take over the public address. A single load balancer in front of four servers only moves the weak spot from the servers to itself.

• Active-passive pair. Two balancers share a floating IP address that can move between machines, often using VRRP, a standard takeover protocol. With its default 1-second announcements, the standby takes over after about 3 to 4 silent seconds.
• Active-active behind routers. Routers spread packets across several balancers using ECMP, equal-cost multipath routing. When one fails, routers may move some other connections too, so the balancers need consistent hashing to keep most connections on their servers.
• Managed cloud balancers. AWS creates a load balancer node in each Availability Zone you enable, a zone being a separate group of data centres inside one region.
• Several addresses in DNS. Route 53's multivalue answer policy returns up to eight healthy records at random, so clients have somewhere else to try.

Maglev uses the active-active pattern, with routers spreading packets over many Maglev machines. Its paper says consistent hashing and connection tracking, which remembers where each connection went, limit the damage when machines fail.

Three weeks later, a music festival sale brought 80,000 buyers in ten minutes. At 12:07, app-2 froze. Health checks pulled it out in about 20 seconds, and no buyer lost a seat hold, because the holds lived in the shared store.

In our tests on nginx 1.24.0, it retried a timed-out GET on the next server but not a POST, a request that changes data. It also kept sending users to a server that answered every request with a 500 error.

We put open-source nginx 1.24.0, the Ubuntu 24.04 package, in front of four small Python servers with default passive checks. We set proxy_read_timeout, how long nginx waits for a server's reply, to 3 seconds so a frozen server times out quickly. Then we froze app-3 with SIGSTOP, a signal that pauses a process.

A paused server still accepts connections, because the operating system does that, but it never answers. That mimics a hung server, which fails slowly instead of refusing connections at once.

Pause and predict: a buyer's POST to hold a seat lands on frozen app-3. Does nginx quietly retry it on app-4, as it does for a GET?

What just happened? A GET is safe to send twice, so nginx retried it and the buyer only waited. A POST may already have changed something, like holding a seat, so nginx returned the timeout instead of risking a double hold.

That rule is documented: since version 1.9.13, non-idempotent requests are not passed to the next server once sent. Idempotent means repeating it has the same effect as doing it once. If you want POST retries, make the endpoint idempotent first, for example with an idempotency key, a unique ID the client sends so the server can spot a repeat.

The 500 row surprises people. nginx's passive check counts connection errors and timeouts as failures, but not a bad answer, unless you list codes like http_500. Here is the setting we tested. In nginx, an upstream is a named group of servers.

upstream app_next500 {
    server 127.0.0.1:9011;
    server 127.0.0.1:9012;
    server 127.0.0.1:9013;
    server 127.0.0.1:9014;
}

server {
    listen 127.0.0.1:8091;
    location / {
        proxy_pass http://app_next500;
        proxy_next_upstream error timeout http_500;
    }
}

To check it on your own setup, send requests while one server returns 500 and search the error log for upstream server temporarily disabled. With the default setting we found no such line; with http_500 listed we found one.

The last row is the opposite of AWS's fail-open. Once every server was marked failed, nginx answered 502 and logged no live upstreams until fail_timeout ran out. The docs say only that $upstream_status records 502 when no server can be selected. They do not say how long that lasts, as of 19 October 2026, so pin your version and test it.

One documented exception: if a group has only one server, nginx never marks it unavailable. So a single-server upstream hides nothing, and its failures go straight to users.

All of the above used one nginx worker process. Workers are the separate nginx processes that handle requests. With four workers and app-3 stopped, each worker logged its own temporarily disabled line, so each one paid separately to discover the failure.

Adding a zone directive, which keeps the group's run-time state in memory shared by all workers, cut that to a single line in both of our runs. If you run several workers, add a zone so they learn about a failure together.

We set HAProxy 2.8.16 to probe every 10 seconds with a 2-second timeout, 2 failures to leave and 3 passes to return. Over 12 runs it removed a frozen server after 14 to 24 seconds. It took the server back 20 to 30 seconds after it recovered.

That is a little slower than the team's model. The timings fit one rule: HAProxy waits 10 seconds after each probe finishes, not after it starts. We inferred that rule from the timings, not from HAProxy's code. So a timed-out probe adds its 2 seconds to the gap. The first failed probe starts 0 to 10 seconds after the freeze. The second gives up 14 seconds after the first one started, so removal lands between 14 and 24 seconds.

Coming back needs three passes in a row, 10 seconds apart, so it takes 20 to 30 seconds. Our first four runs all took 30 seconds only because the test resumed the server right after a failed probe. When we varied that moment, recovery took 20 to 30 seconds, as the rule predicts.

Now apply the same arithmetic to AWS's defaults: a 30-second interval, a 5-second timeout, 2 failures out and 5 passes in. Per load balancer node, a frozen target keeps getting some traffic for about 35 to 65 seconds. After it recovers, it waits about 2 to 2.5 minutes to return.

Those AWS numbers are our arithmetic from the documented defaults, not a measurement. One more AWS detail changes the load on your health endpoint. Each load balancer node checks every target itself, so more enabled zones mean more probes.

Here is the HAProxy backend we tested. The check keyword turns on active checks, and default-server sets the timer and both thresholds for every server.

backend app
  balance roundrobin
  option httpchk GET /healthz
  timeout check 2s
  default-server inter 10s fall 2 rise 3
  server app-1 127.0.0.1:9021 check
  server app-2 127.0.0.1:9022 check
  server app-3 127.0.0.1:9023 check
  server app-4 127.0.0.1:9024 check

To check it yourself, pause one server with kill -STOP and its process ID, then watch HAProxy's log. Expect a line like Server app/app-3 is DOWN, reason: Layer7 timeout, check duration: 2001ms. After kill -CONT, an UP line follows.

A lot, once servers get busy. In our simulation at 85 percent load, round robin's median request took 49 ms against 20 ms for least connections. Its p99, the time that 99 percent of requests beat, was 651 ms against 464.

The simulation is a made-up workload, not a benchmark. Four equal servers each run 8 workers. 80 percent of requests take 20 ms and 20 percent take 400 ms, arriving at random. We ran 200,000 requests five times per algorithm on Python 3.13.

Pause and predict: random choice and round robin both ignore how busy each server is. Which one does worse when the servers are nearly full?

What just happened? At 60 percent load, the median request never waited: 20 ms is just its own work, so the choice barely matters. At 85 percent, round robin keeps feeding a server stuck on seat holds, so quick page loads queue behind them. Random did worse still.

Power of two choices came within 15 ms of a full scan at p99, while looking at only two servers. It checks two servers however many you have, while a full scan checks every one.

Place it in your diagram early, then explain your choices in order: layer, algorithm, health checks, state and redundancy. Tie each choice to a failure it prevents.

1. 1

   Draw it in front of a pool

   Show clients reaching one address, with a pool of identical servers behind it. Say why: spread load, hide failures and add servers easily.

2. 2

   Pick the layer

   Choose L4 for raw speed or non-HTTP traffic, and L7 when you need path or cookie routing. Mention that many designs put L4 in front of L7.

3. 3

   Pick the algorithm

   Use round robin for similar requests, least connections or power of two choices when request times vary, and hashing only when a key must stick to one server.

4. 4

   Describe health checks

   Give an interval, a timeout and both thresholds, then work out how long a dead server keeps getting traffic. Mention draining for deploys and what happens when every server fails.

5. 5

   Handle state and retries

   Keep sessions in a shared store so servers are stateless. Say which requests are safe to retry: reads yes, seat holds only with an idempotency key.

6. 6

   Remove the single point of failure

   Run balancers in a pair or as a managed service. Add global routing across regions if the requirements ask for it.