Latency vs throughput: how fast, and for how many

“Which two words did we mix up on the list?” Meera Kulkarni asks. Arjun Nair has looked them up: “fast” is latency, and the crowd has a word of its own. Then he proudly shows his new chart: the average booking tap takes about 210 milliseconds.

Meera opens her phone. Two club members say the app “spins for three seconds, then says try again”. The chart says fine; the members say broken.

They pull up the saved timings of the last 100 taps on the ₹1,200-a-month server. 98 are quick. 2 took 3 seconds, and those are the messages. Arjun adds a goal to the wall list: “99 out of 100 taps should finish in under 2 seconds.” Today, only 98 do.

Meera frowns. The list has a speed line for one person, and nothing for many people at once. “How many people will tap at noon?” Arjun does not know.

Latency is how long one request takes, and throughput is how many requests finish each second.

People often count throughput in QPS.

• Latency is one request’s wait, adding up hop by hop, the legs of the round trip from lesson 2.
• Throughput is how many finish each second, across everyone.
• The slowest 1 in 100 is what people remember, so watch it.

Here is a different, later SeatJaldi tap as a chain of hops, timed in ms. Hover each box; the times are made up for teaching and differ from lesson 2’s.

You have seen this when a news app opens in a blink on home Wi-Fi but spins on the train. The same hops ran, but each took longer.

A ticket app is quick nearly every time, yet you call it slow after one tap hangs. That rare tap is the one you remember.

On match night, a food app feels slow at every step. Orders arrived faster than it could finish them, like cars at the toll on a festival Friday.

p99 latency is the time that 99 out of 100 requests finish within, so it shows how bad the slowest few are. Line up 100 request times from fastest to slowest; p99 is the 99th one.

Engineers usually watch three percentiles: p50, the middle request; p95, the time 95 in 100 finish within; and p99. The slow few at the far end have a name.

Memory is a thousand or more times faster than a disk or a network trip, and these numbers are orders of magnitude, not exact facts. Memory is the fast working space a running program uses; a disk keeps data when the power is off. An order of magnitude is a rough size, to the nearest power of ten.

A nanosecond (ns) is a billionth of a second, and a microsecond (µs) is a millionth. An SSD is the storage chip inside most laptops. Big apps keep their servers together in one building.

Two well-known tables disagree on the SSD row by about ten times, 150 µs against 16 µs. Picture each row as a longer road. Memory is a walk across the room; a trip between continents is the Bengaluru to Mysuru drive, many times over.

Little’s law ties them together: on average, the number of requests in progress equals how many arrive each second times how long each one takes.

On the expressway, cars on the road equal cars entering each hour times the hours each one drives. At 1,000 cars an hour and 3 hours each, about 3,000 are on the road. If an accident makes every trip 6 hours, 6,000 are, and the jam feeds itself. Requests pile up the same way.

Try it yourself

1. 1

   Find what is in flight

   A site gets 1,000 requests a second, each taking 0.2 seconds (both assumed). How many are in progress at once?

2. 2

   Slow it down

   Now each request takes 0.4 seconds. How many are in progress?

3. 3

   Count the servers

   Assume one server holds 50 requests at once. How many servers in each case?

4. 4

   Check your answers

   You got it if you have 200, then 400, then 4 and 8 servers. When each request takes twice as long, twice as many are held at once.

Measure p99 from the phone’s side, over every request. To find it cheaply, count requests into time buckets, such as under 100 ms, 100 to 300 ms and 300 ms to 1 second. Then find the bucket where the 99th request in 100 lands. It is only as exact as the bucket edges, so let buckets grow about three times each, as the SRE book suggests.

The 0.8-second guide for time to first byte from lesson 2 is the yardstick here. SeatJaldi’s made-up 285 ms fits well inside.

Lesson 2’s curl command also prints the DNS time if you add %{time_namelookup}. Run it 20 times and sort the totals to see your own tail.

Arjun’s chart now shows p99 beside the average, and the angry messages make sense. They were the tail, the two cars behind the accident. Why those two were slow is still open, and a tail needs a cause as well as a number.

“So fast is latency, and the crowd is throughput,” Meera says. “One is a person’s wait. The other is the crowd’s count. Staying up through it is still the availability line we wrote.”

But the wall still has no number for the crowd. Meera can describe speed now. How many requests per second will the noon sale bring?