API Latency Budgets: Estimating Response Time

Each part passes its own tests and looks quick, but together the parts take far longer than the promise. Nobody notices until real users wait. API latency estimation catches this on paper, before anyone writes code.

Picture a common setup. A freight marketplace runs a quote API for small manufacturers who hire trucks and tempos (small goods carriers). A factory manager enters the pickup and drop pincodes (postal codes) and the load weight, then taps Get quote.

The app then shows prices from three transport partners side by side. The product team promised the price within half a second. The setup is made up, but the problem is real.

In the first month, the dashboard showed a median, the middle value, of 380 ms. That looked fine. The p99, the time 99 out of 100 quotes beat, was 1.3 seconds (made up).

So one quote in every hundred took longer than 1.3 seconds. One in a hundred is not rare at this size. The team serves about 40,000 quotes a day, so around 400 quotes a day were slow. Some managers gave up and phoned a broker instead.

The team could have known before writing any code. This guide estimates the quote API on paper, finds the slow hops, fixes the design, and shows you how to check each number yourself.

API latency estimation is working out, before you build, how long one request will take. You list every hop the request passes through, put a time on each, add them up the right way and compare the total with your target.

Four words carry the rest of this guide.

• Latency: the time one request takes, from the moment the caller sends it to the moment the full reply arrives. It is measured in milliseconds (ms), thousandths of a second.
• Hop: one place the request passes through or waits on. Examples are the mobile network, a load balancer (which spreads requests across servers), your service, a database or an outside API.
• Latency budget: your total target split into an allowance for each hop. If the maps call gets 120 ms out of 500 ms, every other hop shares the remaining 380 ms.
• Round trip time (RTT): the time for a message to reach another machine and for an answer to come back. Every full handshake costs at least one.

Think of a tea estate in Darjeeling promising a dispatch date to a buyer in Kolkata. The manager adds up every stage the leaf passes through: plucking, weighing, withering, rolling, drying and packing.

Each stage is a hop. The dispatch date is the target. The date she promises must still hold in a damp week, when withering runs slow, which is the idea behind a percentile target.

Where the picture breaks: tea stages take hours and vary a little. API hops take milliseconds, and the slowest 1 percent of calls can take several times the typical time. In software, the slow tail dominates far more than it does in the garden.

Estimation will not give you the exact number. It gives you a number close enough to show whether the design can work, and which hop to worry about first.

p50 is the time that half of your requests beat, and p99 is the time that 99 out of 100 requests beat. Tail latency means the slowest few requests, usually the ones beyond p99.

A percentile tells you where a value falls once you sort all your timings. Sort one day of the quote API's 40,000 timings from fastest to slowest. With the simplest method, called nearest rank, p50 is the timing at position 20,000 and p99 is at position 39,600.

Monitoring tools use slightly different methods between ranks, so their numbers can differ a little. The idea stays the same.

The median, another name for p50, was 380 ms. That sounds fast. But it says nothing about the 400 slowest quotes at the end of the sorted list, and those are the managers who phoned a broker.

Google's Site Reliability Engineering book makes the same point. It says averaging latencies can hide a long tail of much slower requests. It calls a high percentile, such as the 99th or 99.9th, a plausible worst-case value.

So write your target with a percentile in it. The team's new promise reads: 99 percent of quotes show a price within 500 ms, measured on the phone. A promise like this is a service level objective (SLO), a target you commit to meeting, checked against a measured number.

List every hop in the order the request meets it, write down a time for each, then add up the ones that happen one after another. For calls that run at the same time, count only the slowest.

Here is the path of the team's first version. Follow the arrows from the tap to the price on screen. Notice that each box waits for the one before it, including the three partner calls.

Now give each hop a p99 time. The team fills in the table below from quick tests and the partners' own documents. Read down the last column to see why each number is what it is.

The sum is 1,190 ms, and the dashboard said 1.3 seconds. A paper estimate would have been close enough to stop the launch. It also shows where the time goes: the maps and partner calls take 980 ms of it.

Before anything exists, you rarely need exact figures. A rough number from any of these places is enough for a first estimate.

• Outside APIs: their published response times, or a few hundred test calls from a server in your cloud region.
• The network: a ping from a phone on a mobile network to your cloud region, at a busy hour.
• Databases: a timed query against a copy of realistic data, not an empty table.
• Your own code: often a few milliseconds, unless it does heavy work such as image processing. Time it once to be sure.
• Services you already run: their dashboards are the best guide to what a similar hop will cost.

A brand new HTTPS connection over TCP with TLS 1.3 usually costs about three round trips before the first byte of a small reply arrives, not counting the address lookup. One is for the TCP handshake, one for the TLS handshake and one for the request itself.

TCP is the protocol that carries most web traffic. RFC 9293, the TCP standard, opens a connection with a three-way handshake. The client sends SYN, a request to start. The server answers SYN-ACK, agreeing, and the client replies with ACK. That costs one round trip before the request can go.

TLS is the layer that encrypts the connection, the S in HTTPS. A full TLS 1.3 handshake, defined in RFC 8446, needs one round trip. A full handshake in the older TLS 1.2 needs two. Only then does the encrypted request go out, which is the third round trip.

From a phone in Gurugram to the team's cloud region, the round trip is about 60 ms (made up). So the first version paid 60 ms three times, 180 ms, on every quote. Inside the data centre it also opened a fresh connection to the maps API on every call.

That added two more round trips of about 30 ms each, 60 ms in all, to the maps hop. Two other costs sit outside this count. A phone with no cached DNS answer must first look up your server's address. DNS is the internet's address book, turning a name like api.example.com into the numeric IP address computers use. A large reply also needs more round trips to arrive.

The diagram shows the choice every request faces. Start at the tap and follow the No arrow to see the extra handshakes. The Yes arrow is the shortcut that connection reuse gives you.

The fix is to reuse connections. HTTP/1.1 keeps a connection open after each response by default, which RFC 9112 calls a persistent connection. People also call it keep-alive. Only the first request on it pays for the handshakes.

The quote service now keeps a pool of open connections to the maps API and to each partner. On the phone, the app opens its connection as soon as the quote screen loads. While the manager types two pincodes and a weight, the handshakes finish.

Servers and load balancers close connections that sit idle for too long, each with its own timeout. So a connection opened long before the tap may need to be opened again. Check the idle timeout on your load balancer.

When a client reconnects to a server it has spoken to before, TLS 1.3 offers a 0-RTT mode that sends some request data in the very first message. RFC 8446 warns that this early data has no guarantee against replay, where an attacker resends a captured message. It is also not forward secret: someone who steals the key later could read it. Use it only for requests that are safe to repeat.

Parallel calls cost as much as the slowest one, and each extra call is one more chance to be slow. So the more services a request waits on, the more often it lands in the tail.

Fan-out means one request sends calls to several services and waits for their answers. The quote API fans out to three transport partners on every quote. The obvious first fix is to call them at the same time instead of one by one.

Back at the tea estate, three plucking groups work three slopes at once. The lorry to the factory leaves only when the last group's baskets are weighed. One slow group on a steep slope delays the whole load.

In the diagram, look at the box in the middle. It waits for all three partners, so its time is the slowest arrow, not the sum of them.

Parallel calls fix the sum, but not the tail. Pause and predict: say each partner is slow on 1 call in 100, independently of the others. What share of quotes now waits on at least one slow partner?

A quote is fast only if all three partners are fast. That chance is 0.99 times 0.99 times 0.99, about 0.970. So about 3 percent of quotes wait on a slow partner, three times the rate of one partner alone.

What just happened? The quote's p99 is no longer a typical partner time. With 3 percent of quotes slow, the slowest 1 percent of quotes all sit inside the slow cases. A simulation confirmed the 3 percent.

We also simulated three partners with the same made-up spread, each with a p99 near 400 ms. The slowest of the three had a p99 near 500 ms. Same parts, slower whole.

Jeffrey Dean and Luiz Barroso described this effect in their 2013 paper, The Tail at Scale. In their example, a server takes over one second on 1 request in 100. A user request that waits on 100 such servers takes over one second 63 percent of the time.

They also measured a real service at Google that fans out to many servers. One request's p99 was 10 ms, but the p99 for all requests to finish was 140 ms. Waiting for the slowest 5 percent of requests made up half of that total.

You cut latency by changing the design, not by tuning code. Remove hops, reuse connections, cache slow lookups, run independent calls in parallel, and cap slow calls with a timeout and a fallback.

The team takes the table and goes through it hop by hop. Each change below targets one row.

• Reuse connections. The phone opens its connection early, and the service pools connections to the maps API and partners. The network hop drops from 180 ms to 60 ms.
• Cache the rate card. Rates change once a day, so the service keeps them in memory and reloads every hour. That hop drops from 15 ms to about 1 ms.
• Call partners in parallel. The three calls now cost the slowest one, not the sum.
• Cap each partner at 200 ms. A partner that has not answered by then is left out of this quote.
• Cap the maps call at 120 ms. If it is late, the service estimates road distance from straight-line distance and marks the price as approximate.

The team also added a cache of distances between pincode pairs. It answers 97 percent of quotes, which makes the median far faster. The section on caches below shows why it alone would not fix the p99, and why the timeout does.

It usually overshoots a little, because independent hops are rarely all slow at the same moment. It fails in two ways: when hops share a cause of slowness, and when each hop has rare stalls beyond its own p99.

We tested this with a simulation of the first version's eight hops, each with a made-up spread of timings. Summed p99s gave 1,191 ms. The p99 of the simulated total was 987 ms, about 17 percent lower.

Then we made every hop slow at the same moment, as if one shared cause, like a busy network, slowed them all. The p99 of the total rose to 1,195 ms, matching the sum. Shared causes remove the safety margin.

The second failure is worse. We simulated eight hops that each take about 10 ms but stall for 500 ms on 0.5 percent of calls. Each hop's p99 is about 12 ms, so summed p99s gave 96 ms.

Yet about 4 percent of totals hit at least one stall, the same fan-out maths as before. The real p99 of the total was 572 ms. A per-hop p99 can hide a stall that the whole request still feels.

Now look back at the quote API. Its paper sum was 1,190 ms, yet the dashboard showed 1.3 seconds. The gap came from things not on the list: DNS lookups, retries and queueing at busy hours (made up). A sum only covers the hops you wrote down.

Because the p99 is decided by the slowest 1 percent of requests, and cache misses are the slow ones. Here every miss is slower than every hit. So when more than 1 percent of requests miss, the slowest 1 percent are all misses, and the p99 still includes a full call.

The pincode cache answered 97 percent of quotes. In a simulation with that hit rate, the median maps time fell to 1.0 ms, and the p95 (the time 95 in 100 beat) to 1.5 ms. Without the cache, the maps p99 was 310 ms. With it, the p99 was 217 ms: lower, but still a full maps call, not the 1.5 ms of a hit.

It fell because the slowest 1 percent of quotes are now the slower third of the misses. Raising the hit rate to 99.5 percent moved the p99 to 1.5 ms, because misses were now rarer than 1 percent. So a cache brings the p99 down to hit speed only when misses are rarer than the percentile you promise.

That is why the timeout protects the quote API's p99. In the simulation, almost every miss took longer than 120 ms, so about 3 in 100 quotes, some 1,200 a day, would show an approximate price. The product team accepted that, because the exact fare is confirmed when the manager books.

A hedged request sends the same request to a second replica, a copy of the service, when the first is slow to answer. You use whichever reply arrives first and cancel the other.

The Tail at Scale suggests waiting until the first request has waited longer than the 95th-percentile expected time before sending the second. So only the slowest 5 percent or so of requests get a second copy, which keeps the extra load small.

In the paper's benchmark, a read of 1,000 keys spread across 100 servers used a 10 ms hedging delay. It cut the 99.9th percentile from 1,800 ms to 74 ms, while sending only 2 percent more requests.

Hedge only requests that are safe to repeat: reads such as a price, or writes that carry an idempotency key, an ID the server uses to ignore duplicates. Never hedge a booking or a payment without one, because both copies could succeed. Retries with backoff follow the same rule about requests that are safe to repeat.

Write the target as a percentile, draw the request path, give every hop an allowance, and check that the allowances plus some spare fit inside the target. Then give each allowance an owner and an alert.

1. 1

   Write the target as an SLO

   Name the percentile, the number and where it is measured. The team's reads: 99 percent of quotes show a price within 500 ms, measured on the phone.

2. 2

   Draw the request path

   List every hop from the user's device to the reply, including handshakes and every outside API. Mark which calls run one after another and which run together.

3. 3

   Put a p99 on every hop

   Use quick tests, published numbers or dashboards from similar services. Rough numbers are fine at this stage.

4. 4

   Add it up the right way

   Add hops that run in a row. For calls that run together, take the slowest one. Then check the chance that one of them is slow.

5. 5

   Leave some spare

   Keep part of the budget unused for what you forgot, such as DNS lookups, retries and busy hours. The team keeps about 20 percent.

6. 6

   Give every hop an owner

   Each allowance becomes a number a team watches, with an alert when the hop's p99 goes over it.

The team keeps the budget next to the code in a small YAML file, a plain text format for settings. A change to it gets reviewed like any other change. A parallel entry lists calls that run at the same time.

budget_ms: 500            # p99, measured on the phone
path:
  - {name: mobile round trip, ms: 60}         # warm connection
  - {name: load balancer and gateway, ms: 10}
  - {name: quote service code, ms: 5}
  - {name: rate card from memory, ms: 1}
  - {name: maps distance, ms: 120}            # timeout, then fallback
  - parallel:
      - {name: partner A, ms: 120}
      - {name: partner B, ms: 150}
      - {name: partner C, ms: 200}            # capped by the timeout

A short Python script reads the file and checks it. It uses the PyYAML library, which you install with pip install pyyaml. We ran it on Python 3.13 with PyYAML 6.0.3.

import sys
import yaml

def step_ms(step):
    if "parallel" in step:
        return max(step_ms(s) for s in step["parallel"])  # slowest one wins
    return step["ms"]

def check_budget(path_file):
    with open(path_file) as f:
        plan = yaml.safe_load(f)
    total = sum(step_ms(s) for s in plan["path"])  # steps in a row add up
    spare = plan["budget_ms"] - total
    print(f"Estimate {total} ms, budget {plan['budget_ms']} ms, spare {spare} ms")
    return spare >= 0

def chance_any_slow(p_slow, calls):
    return 1 - (1 - p_slow) ** calls  # at least one call lands in the tail

if __name__ == "__main__":
    print(f"Chance one of 3 partners is slow: {chance_any_slow(0.01, 3):.4f}")
    sys.exit(0 if check_budget(sys.argv[1]) else 1)  # non-zero exit fails the build

The line with max is the parallel rule: a group of calls costs its slowest member. The line with sum is the in-a-row rule. Save the script as check_budget.py and the settings as latency_budget.yaml, then run python3 check_budget.py latency_budget.yaml. On some systems the command is python instead.

You should see two lines: a 0.0297 chance that one of three partners is slow, then an estimate of 396 ms with 104 ms spare. The exit code is 0. Lower the budget to 350 ms and it prints spare -46 ms and exits with 1.

That exit code is what makes it useful in a build. A change that adds a hop and breaks the budget fails the build before it ships.

Measure the same percentile at the same place your target names, then compare each hop's real p99 with its allowance. Where the two disagree, either the estimate or the design needs fixing.

Count your handshake round trips with curl

You can see the round trips of one request with curl's timing variables. Each one is in seconds, counted from the start of the request. Run this against an endpoint of your own API, replacing the address.

curl -s -o /dev/null -w 'connect=%{time_connect}s tls_done=%{time_appconnect}s first_byte=%{time_starttransfer}s\n' https://api.example.com/health

connect includes the DNS lookup and the TCP handshake. The gap from connect to tls_done is the TLS handshake. The gap from tls_done to first_byte is the request round trip plus your server's time.

We ran this in October 2026 with curl 8.5.0 and OpenSSL 3.0.13. A local relay added 30 ms each way to the TLS and HTTP messages, so each round trip was 60 ms. The TCP handshake stayed local, so connect read near zero.

The table rounds the ranges from several runs on the same versions. Your times will vary with CPU load, but the round-trip counts should match. A few runs on an overloaded machine were much slower, one request taking 376 ms, and are left out.

On a real network, add one more round trip for the TCP handshake inside connect. If your TLS gap is close to two round trips, check the version: curl -v prints a line such as SSL connection using TLSv1.3. A TLS 1.3 server that asks the client for a different key share, called a HelloRetryRequest, also adds a round trip.

Trace every hop and time it on the device

Distributed tracing is the main tool for per-hop numbers. A trace follows one request through every hop and records how long each one took. The team can open a slow quote and see partner C cut off by its 200 ms timeout.

Measure on the device too. The app records the time from tap to price and sends it with the next request. Server dashboards alone would never have shown the 180 ms of handshakes on the mobile network.

Test at your busiest hour

As a server gets close to full, requests wait in a queue for their turn. Take the textbook single-server queue: requests arrive at random, and each needs a random amount of work, 10 ms on average. The average time per request is 20 ms at 50 percent busy.

At 90 percent busy, that average becomes 100 ms, five times higher, and the p99 is about 460 ms. If every request needed exactly 10 ms of work, the 90 percent average would be 55 ms: lower, but still over five times the work. So test at the busiest hour, not on a quiet afternoon.

Three months after the redesign, the phone-side p99 is 430 ms (made up). That is higher than the 396 ms estimate, but inside the 500 ms promise. The spare did its job.

Explain it as a sum you do out loud. State the target as a percentile, walk the request path, put a number on each hop, and show which design choices make the total fit.

1. 1

   Pin down the target

   Ask for the percentile and where it is measured. Suggest one if the interviewer has none, such as p99 under 500 ms on the device.

2. 2

   Walk the path

   Name each hop from the client to the reply: DNS, handshakes, load balancer, service, database and every outside API.

3. 3

   Put rough numbers on each hop

   Use round figures and say they are guesses. Say how many round trips a new connection costs, and what one round trip costs here.

4. 4

   Add up the right way

   Sum the hops in a row, take the slowest of the parallel calls, and mention that fan-out makes the tail more likely.

5. 5

   Fix the biggest rows first

   Propose connection reuse, caching, parallel calls and timeouts with a fallback. Say what the user sees when a timeout fires.

6. 6

   Close with measurement

   Describe tracing per hop, timing on the device and a load test at peak traffic to check the estimate.