LLM Latency Optimization: Accuracy vs Latency Trade-offs

A correct answer that arrives late can still fail the person waiting for it. That is the LLM accuracy vs latency problem in one line. In a voice assistant, silence feels long, and the listener is often doing something risky, like driving.

Picture a common setup. A ride-hailing app in Bengaluru adds a voice assistant for its drivers. A driver presses one big button and asks, "Where exactly is my rider waiting?" The setup is made up, but the problem is real.

Before the assistant, a driver who could not find a rider had two choices. Pull over and tap through the app, or call the helpline and wait on hold. With made-up numbers, each missed pickup costs a Rs 30 refund, and 300 a day cost Rs 9,000.

The first version of the assistant gave correct answers. But in the team's made-up timings, the driver heard the first word about 5 seconds after they stopped talking. At 40 km per hour, a cab covers about 55 metres in 5 seconds.

Drivers in the test group kept glancing at the screen to check it was still working. So the team set a goal: on common questions, the first word in about 1 second. This guide follows how they got there, and where they chose to stay slow on purpose.

You need six words to follow this guide. Each one gets a plain meaning and an example from the driver assistant.

Latency. Latency is waiting time: how long from a request until the answer arrives. Example: the driver stops speaking, and 5 seconds later the assistant starts talking. That 5 seconds is the latency they feel.

Token. A token is a small chunk of text that the model reads and writes, often a short word or part of a longer word. Example: "Gate 2 of the tech park" is several tokens. The model writes its answer one token at a time.

Time to first token. Time to first token, or TTFT, is how long until the first token of the answer comes back. Before writing anything, the model must read the whole prompt. A longer prompt usually means a longer TTFT.

Output speed. Output speed is how many tokens the model writes per second once it has started. With made-up numbers: at 50 tokens a second, a 110-token answer takes 2.2 seconds to write. Every extra word is extra waiting.

Percentile (p50 and p95). Sort your requests from fastest to slowest. The p50 is the middle one: half of requests are faster. The p95 is the time that 95 in 100 requests take or beat, so at most 5 in 100 are slower.

Accuracy. Here, accuracy means how often the answer is correct and useful for the person asking. Example: "Your rider is at Gate 2" is accurate only if she really is at Gate 2 right now. A fast wrong answer can cost more time than it saved.

You balance LLM accuracy vs latency by giving each kind of question its own time budget and risk level. Common, low-risk questions get a fast path. Questions about safety or money get a slower, checked path.

A latency budget is the total time you allow, split across the steps, like a monthly household budget split across rent, food and travel. The team's budget: 1 second to the first word for common questions, and 2 seconds at p95.

To spend a budget well, you need to know where the time goes. Think of a darshini, the stand-up restaurants all over Bengaluru. Your wait for a masala dosa has parts, and each part maps to a part of an LLM request.

• The queue at the counter is the network trip and the time your request waits for a free server.
• The cook reading your order slip is prefill: the model reading the whole prompt before it can start.
• The cook making the dosa is decode: the model writing the answer, token by token.
• The counter handing over each dosa as it comes off the pan is streaming: you start eating before the whole order is ready.

Where the picture breaks: a cook can pour two dosas side by side, but with plain decoding, one answer's tokens come strictly one after another. A model server does work on many people's requests at once, but that is a separate trick called batching.

LLM latency comes from every step between the question and the answer, not only from the model. The model's reading time, its writing time, network trips, searches through your data and tool calls all add up.

The team's first move was to change nothing. They added a timer around every step and watched one question travel through. Follow the boxes from left to right and read the time in each note. The numbers are made up, but shaped like real ones. To keep the sums simple, network trips are counted inside each box, and the wait for the voice to start speaking is left out. Real voice budgets need both.

Here is what each part means:

• Network: each box that talks to another service pays a trip there and back. On a phone in a moving cab, these trips can be slow and uneven. Reusing one open connection and calling servers close to your users both help.
• Retrieval: searching your own data for useful notes, like "Riders at this tech park wait at Gate 2, not the main gate." Bigger searches and slow databases add time.
• Tool calls: asking another system to do something, like fetching the rider's live location. When calls run one after another, their times add up.
• Time to first token: the model reads the whole prompt before writing. Long instructions and many pasted notes make this slower.
• Output length: the model writes one token after another, and each token depends on the ones before it. At a steady speed, 110 tokens take almost four times as long as 30.

Look at the biggest box: 2.2 seconds just to write the answer. The first version gave long, polite answers like "Your rider has shared her location, and according to the latest update..." The driver only needed one short sentence.

You measure LLM latency by recording the time of every real request, sorting the times, and reading the middle value (p50) and the slow-end value (p95). Do this separately for the first word and for the full answer.

Pause and predict. Say 94 of 100 questions take 1 second and 6 take 10 seconds. What is the average, and what is the p95?

The average is 1.54 seconds, which looks fine. The p95 is 10 seconds, which shows that 6 drivers in every 100 sat in silence. A few very slow requests hide inside a nice-looking average.

The assistant saves a small timing record for each question: when the driver stopped talking, when the first word played, and when the last word played. The code below computes the numbers from the example above, using Python 3 and no libraries.

def percentile(values, p):
    ordered = sorted(values)
    index = round(p / 100 * (len(ordered) - 1))
    return ordered[index]

first_word = [1.0] * 94 + [10.0] * 6    # seconds, the example above

print("average:", round(sum(first_word) / len(first_word), 2), "s")
print("p50:", percentile(first_word, 50), "s")
print("p95:", percentile(first_word, 95), "s")

Run it and you should see "average: 1.54 s", "p50: 1.0 s" and "p95: 10.0 s". What just happened? The function sorts the times and picks the value at the right position: halfway along for p50, 95 percent of the way along for p95.

In production, feed it the first-word times and the full-answer times from your logs, as two separate lists. Libraries use slightly different rules for picking the position, so small samples can differ a little. With made-up numbers, the first version showed a first-word p50 of 4.9 seconds and a p95 of 8.2 seconds.

To check your own TTFT, time the gap until the first non-empty piece of a streamed answer. The helper below works on a plain loop that yields text pieces, not an async one (code that waits without freezing the program). Start the request inside that loop, so the timer covers the whole wait. Here it runs on a fake stream that waits 0.8 seconds, then writes five words.

import time

def measure(stream):
    start = time.perf_counter()
    first = None
    for chunk in stream:
        if chunk and first is None:
            first = time.perf_counter() - start
    total = time.perf_counter() - start
    print(f"time to first token: {first:.2f} s")
    print(f"time to full answer: {total:.2f} s")

def fake_stream():
    time.sleep(0.8)              # the model reads the prompt
    for word in ["She ", "is ", "at ", "Gate ", "2."]:
        yield word
        time.sleep(0.02)         # the model writes the next token

measure(fake_stream())

Run it and you should see "time to first token: 0.80 s" and about "time to full answer: 0.90 s". To measure the real thing, write a small generator (a function like fake_stream()) that sends the request and yields each streamed piece, and run it from your own network. If the request starts before measure() runs, the timer misses the wait and can show almost 0 seconds. For an async client, time it the same way inside an async for loop.

Streaming makes an LLM feel faster by sending each piece of the answer as soon as the model writes it. The time to the full answer stays about the same, but the person starts hearing or reading much sooner.

Back at the darshini, if you order three dosas, the counter hands over each one as it comes off the pan. Nobody waits for the third dosa to start eating. Streaming works the same way: the first sentence goes out while the model writes the second.

Most model services can stream, and chat apps use this to show text appearing word by word. For voice, the team waits for one full sentence, then sends it to text to speech, the software that speaks text aloud.

With made-up numbers, streaming moved the first word from 5.0 seconds to about 3.0. The first sentence of about 10 tokens took only 0.2 seconds to write. The rest kept arriving while the driver was already listening. But streaming has an accuracy cost.

The best LLM latency optimization techniques are shorter outputs, smaller models for simple steps, running independent calls at the same time, and caching. Pick them in order of which step is slowest in your own timings.

Ask for shorter answers

Shorter answers are usually the cheapest win, because writing time grows with every token. The team changed the instructions to: "Answer in one short sentence a driver can act on. No greetings." They also set a limit of 40 output tokens.

Most model services and vLLM accept such a limit, often called max_tokens. It is only a safety net: if the answer hits it, the answer is cut off mid-sentence. The average answer fell from 110 tokens to about 30, so writing time fell from 2.2 seconds to 0.6.

Use a smaller model for simple steps

A smaller model has fewer parameters (the numbers it learned in training). On the same hardware it usually starts and writes faster, but it makes more mistakes on hard tasks. Use it only for steps that are simple and easy to check.

In the first version, the big model did the labelling step: is this a pickup, route, payment or safety question? That took 0.6 seconds. The team moved this one step to a small model, and it dropped to about 0.15 seconds. The big model still writes the answer.

Run independent calls at the same time

If two steps do not need each other's results, start both together. Then you wait only for the slower one, instead of both added together.

Searching pickup notes (0.4 seconds) and fetching the rider's location (0.7 seconds) do not depend on each other. Run one after the other, they take 1.1 seconds. Run together, they take 0.7 seconds. In the diagram, the two middle boxes start together and meet again before the model.

The labelling step still comes first, because its result decides which searches and tools to run. Add it up with made-up numbers: 0.3 + 0.15 + 0.7 + 0.8 + 0.2 for the first sentence. The first word now arrives after about 2.15 seconds, down from 3.0.

Here is the new flow in Python. The helpers get_rider_location, search_pickup_notes, build_prompt, stream_sentences and speak are made up for this guide. stream_sentences sends a prompt to the model and hands back the answer one full sentence at a time.

import asyncio

async def answer_pickup_question(question, ride_id):
    location, notes = await asyncio.gather(
        get_rider_location(ride_id),      # tool call, about 0.7 s
        search_pickup_notes(ride_id),     # retrieval, about 0.4 s
    )
    prompt = build_prompt(question, location, notes)
    async for sentence in stream_sentences(prompt, max_tokens=40):
        await speak(sentence)             # the driver hears it right away

What just happened? The async and await keywords let Python wait for a slow call without freezing the program. asyncio.gather starts both calls together and waits until both finish, so the wait is about 0.7 seconds, not 1.1.

The async for loop then receives the answer one sentence at a time and speaks each one straight away. With stand-in helpers that sleep for the made-up times, the data wait measured 0.70 seconds with gather and 1.10 seconds one after the other.

Some research goes further and writes parts of one answer in parallel. One paper first asks the model for a short outline, then expands every point at the same time. Tested on 12 models, it was more than 2 times faster on 8 of them, and up to 2.39 times.

The same paper says the method does not suit questions that need step-by-step reasoning or only a short answer. So it can help a long list of tips, not the driver's one-line answers.

Caching cuts latency by saving work you already did and reusing it. Precomputing cuts it by doing the work before anyone asks. Both are fast because the slow work is no longer on the path while the driver waits.

A cache is a store of saved results that your program checks before doing the slow work. Three ideas help here.

• Answer cache: save whole answers to questions whose answer rarely changes. "How do I end a trip?" has the same answer for every driver this month, so it can come back with no model call at all.
• Prompt caching: model services and servers such as vLLM can reuse their work on the start of a prompt that repeats across requests, like the team's fixed instructions. It shortens TTFT, not writing time.
• Precomputed data: fetch what the next question will probably need before it is asked. The moment a driver accepts a ride, the system fetches the rider's location and pickup notes, and refreshes them every 15 seconds.

The team also moved the fixed instructions to the start of the prompt, so prompt caching could reuse them, and cut them to half their size. In their made-up numbers, TTFT fell from 0.8 to 0.4 seconds. With the location and notes already in hand, the data wait drops to zero.

Now the first word arrives after about 1.05 seconds: 0.3 for speech, 0.15 for labelling, 0.4 for the model to start and 0.2 for the first sentence. That is within 50 milliseconds of the 1-second goal.

Precomputing is a guess, and guesses can be wasted. In the team's made-up numbers, most accepted rides never lead to a pickup question, so most background fetches are thrown away. They only precompute where the guess is cheap and often right.

Here is every change side by side. The time saved is the made-up saving on the pickup question. The last two columns are the part people forget: what each change can cost in accuracy, and when it is safe.

After all the changes, the team's dashboard showed, in made-up numbers, a first-word p50 of 1.0 second and a p95 of 2.1 seconds. The p95 was still over budget. Most slow requests came from cache misses (questions the cache had no saved answer for) and weak mobile signal under flyovers.

Accuracy should win when a wrong answer is expensive or hard to undo, such as anything about safety, money or the law. In those cases, a slower right answer is cheaper than a fast wrong one.

One evening a rider tells the driver she already paid by UPI, India's instant bank-transfer system. The driver asks the assistant whether to still collect cash. A stale cache could quickly say, "Yes, collect Rs 340." If that is wrong, the rider pays twice and the company must refund her.

Signs that accuracy must win:

• Safety: accidents, a rider who feels unwell, a road closure, or anything about police or hospitals.
• Money: payments, refunds, fares, penalties, or anything that changes what someone pays or earns.
• Actions that cannot be undone: cancelling a ride, blocking a rider, or reporting a driver.
• Rare questions: if a question type is uncommon, your fast paths and caches were never tested on it.

Slow does not have to mean silent. Within about half a second, the assistant says a holding line, such as "Checking your payment, one moment". Then it reads fresh data from the payment system and uses the big model with full checks.

The driver waits about 2.5 seconds in the made-up timings, but knows it is working, so their eyes stay on the road. The diagram shows how each question picks its path. Notice that plain code checks for safety words before the small model runs, and emergency words go straight to the safety desk.

The plain-code check exists because the small model is fast but not perfect. In the team's made-up test of 500 sample questions, the small model labelled 97 in every 100 correctly, and the big model 99. Three wrong labels in a hundred is fine for pickups, not for "my rider has fainted".

This is also why every speed change needs an accuracy check. Before shipping any fix from the table, the team runs the same 500 sample questions and compares the answers with the old version. If accuracy drops on a question type that matters, the fix does not ship for that type.

Inside the server, the two phases from earlier behave differently, so most speed-ups touch only one of them. Prefill reads all the prompt's tokens together, in large parallel chunks. Plain decoding then writes one token per step for each request.

The version notes below are for vLLM 0.32.0, the open-source model server, released on 18 October 2026 and checked as of 19 October 2026.

Prefill and decode get in each other's way

A research paper on serving design measures TTFT for the prefill phase and TPOT for the decoding phase. It reports that running both phases on the same GPUs, the graphics chips that run most models, causes strong interference between them.

Its system places prefill and decoding on different GPUs, so each phase can meet its own target. For an app team, the lesson is smaller: a long new prompt from one user can slow down the token stream of another user on a shared server.

Prefix caching only speeds up the start

Prefix caching saves the model's work on the start of a prompt and reuses it when a new prompt begins with the same tokens. The vLLM 0.32.0 docs say it only cuts prefill time, not decoding time.

So it helps little when most of the time goes on writing a long answer, and not at all for prompts that share no prefix. Put the fixed instructions first and the changing parts, like the rider's location and the question, last. Changing one early token stops the reuse from about that point on.

A docs-versus-code gap, as of 19 October 2026: the vLLM 0.32.0 prefix caching page tells you to set enable_prefix_caching=True. The 0.32.0 source already turns it on by default for most text-generating models, in vllm/engine/arg_utils.py. Pin your version before relying on either.

One research paper reuses saved work for repeated prompt parts, even when they are not at the very start. On long prompts, such as questions about a document, it reports TTFT improvements from 8 times on GPUs up to 60 times on CPUs. It reports output accuracy comparable to running without the cache, not identical. Each saved part was worked out without seeing the other parts, so the reuse is an approximation.

Speculative decoding speeds up writing, in theory without changing the output

Speculative decoding attacks decode time. A small draft model guesses the next few tokens, and the big model checks all the guesses in one pass. The big model accepts guesses in order until one fails its check, then writes the next token itself.

Two papers described it within three months of each other, in November 2022 and February 2023. One reports 2 to 3 times faster sampling on the large model it tested, with identical outputs. The other reports a 2 to 2.5 times decoding speed-up in a distributed setup, without lowering sample quality.

The acceptance rule is what keeps the answer unchanged. We ran it on a toy model with 400,000 draws: every token's frequency matched the big model's within 0.1 percentage points. The small model alone was far off. A toy greedy run, always picking the most likely token, gave identical text using 11 big-model passes instead of 30.

Real servers add one honest caveat. The vLLM 0.32.0 docs call speculative decoding lossless in theory, up to tiny rounding differences in the chip's arithmetic. They also say changing the batch size can shift output probabilities slightly. The lossless claim holds for vLLM's default acceptance rule, called standard. As of 0.32.0, a setting called synthetic accepts guesses at fixed rates, so it does not keep the output.

The same docs aim it at the gap between tokens when traffic is medium to low, and rate the gains by method and traffic level. Measure TPOT on your own traffic before and after turning it on.

Explain it as a budget. Set a latency target, find where the time goes, fix the slowest steps first, and name what each fix might cost in accuracy. Interviewers want to see that you measure before you optimise and know when not to.

1. 1

   Pin down the target

   Ask which latency matters, first word or full answer, and at which percentile. For the driver assistant: first word in about 1 second at p50, under 2 seconds at p95.

2. 2

   Break the request into steps

   List speech to text, labelling, retrieval, tool calls, time to first token and writing time, with a rough time for each.

3. 3

   Fix the biggest steps first

   Shorter answers and streaming, then parallel calls, then caching and precomputed data for the most common questions.

4. 4

   Name the accuracy cost

   For each fix, say what could go wrong: stale caches, wrong labels from a small model, unchecked streamed sentences.

5. 5

   Protect the risky paths

   Send safety and money questions down a slower, checked path with a quick holding line, and a person for emergencies.

6. 6

   Say how you will measure

   Track p50 and p95 per step, and run a fixed set of sample questions before every change to catch accuracy drops.