Prompt Caching Explained: Faster, Cheaper LLM Calls

The model reads the whole document again for every question, and you pay for that reading each time. Nothing carries over from one call to the next unless something like prompt caching is built to keep it.

Picture a common setup. A legal research tool lets lawyers upload a case file and ask questions about it. One morning a lawyer uploads a 40-page property dispute file and asks 20 questions, one after another. The setup is made up, but the problem is real.

The tool uses a large language model, or LLM: a program that reads text and writes text back. Everything the tool sends to the model in one request is called the prompt. Every prompt here holds the same three things: the tool's instructions, the full case file and the new question.

The question is one line. The case file is 40 pages. With made-up prices, each question costs about Rs 3.42 (Indian rupees) and the lawyer waits 4 seconds before the first word appears. About 94 percent of that bill pays for reading the prompt, and almost all of the prompt is the same 40 pages.

You need five words for the rest of this guide. Each one gets a plain meaning and an example from the case file tool.

Token. A token is a small piece of text the model reads, roughly a short word or part of a word. Model services count prompts and answers in tokens, and hosted ones charge per token. Example: the case file and instructions come to about 32,000 tokens (made up), and a question is about 200.

Prefix. A prefix is the beginning part of something. The prefix of a prompt is everything from the first token up to some point. Example: the instructions plus the case file form a 32,000-token prefix that is the same for all 20 questions.

KV cache. Before the model writes a single word, it reads the whole prompt and builds working notes for every token. Engineers call these notes the key-value cache, or KV cache, after the two kinds of numbers it stores. Example: for each question, the model builds notes for all 32,000 tokens of the prefix.

Cache hit and cache miss. A cache is a store of saved work you can reuse. A hit means the saved work was found and reused; a miss means the work is done from scratch. Example: the second question is a hit if the notes for the case file are still saved.

Time to first token. This is how long you wait until the first word of the answer appears. Example: the lawyer stares at an empty answer box for 4 seconds (made up) before the tool starts typing.

Prompt caching means a model service keeps the KV cache it built for the start of a prompt and reuses it when a later prompt starts the same way. The model then processes only the new part at the end.

For the case file tool, the notes for the 32,000-token prefix come out the same on every question, because the instructions and the case file never change. Prompt caching keeps those notes for a while. On the next question, the service loads them and builds notes only for the 200 new tokens.

Caching does not change what the model computes; it skips repeating work. vLLM's design notes say prefix caching won't change model outputs.

We tested a tiny two-layer model in NumPy, a Python maths library. Cached and fresh outputs differed only in the 15th decimal place, a rounding error. Real GPUs, the graphics chips that run most models, can show tiny differences too, as the FAQ explains.

The diagram follows one question. Look at the split after the cache check: a hit skips the slow reading step, and a miss reads everything and keeps its notes for next time.

Hosted services differ in the details. Check your provider's current docs for three things: whether caching is automatic or needs a marker, the minimum prompt length, and how long a cached start lasts. These rules change, so do not hard-code them from memory.

Because from the model's second layer on, each token's notes depend on every token before it. Change token 5, and those notes change for every later token too, even if the later text is identical. So saved notes are safe to reuse only up to the first difference.

The toy-model test above checks this. Changing token 5 of a 300-token prompt left the second-layer notes for tokens 0 to 4 unchanged. It changed the notes for all 294 tokens from 6 onwards.

A 2025 audit of 17 hosted services notes that providers describe hits as happening only for exact prefix matches. Research systems can reuse other parts with extra care, but this guide sticks to prefix matching.

None of it. To you, one space means nothing. To the cache, the token sequence now differs at the top. The notes for every later token differ too, so all 32,000 tokens are read again.

• Same instructions, same case file, new question at the end: the first 32,000 tokens match, and only the question is read fresh.
• A corrected case file with a change on page 30: everything before the change still matches (rounded down to a whole block in vLLM), and the rest is read fresh.
• A corrected page 1: the match ends almost at once, so nearly the whole case file is read fresh.
• A timestamp or extra space at the top: the match ends there, so the whole case file is read again.

Put the parts that stay the same first and the parts that change last. The longer the unchanging start of your prompt, the more work the service can reuse.

A good order runs from "never changes" to "changes every time". The case file tool's prompt also carries a tool list. A tool is an action the app lets the model ask for, such as searching past court judgments. Follow the diagram from left to right and notice where the cached part stops.

Look at the history box. When a lawyer asks follow-up questions in one chat, earlier turns are added at the end. Old turns stay in the same place every time, so they remain part of the matching start as the chat grows.

A cached start lasts only while the service keeps it, and that depends on the service. Hosted services set and document their own lifetime. vLLM has no timer: it keeps cached notes until it needs the memory for new requests.

Cache lifetime

The saved notes take up memory on the machines that run the model, usually GPU memory, so they cannot be kept forever. Each service decides when to drop them.

Think of a chai stall near the court. While customers keep arriving, the tea on the stove stays hot and each cup is quick. If nobody comes for an hour, the tea goes cold and the next cup is made from scratch.

The map: the stove is the service's memory, the hot tea is the cached notes and each customer is a request. Where the picture breaks: tea cools with time alone, but vLLM drops notes only when it needs the room.

Say the tool's service keeps a cached start for 10 minutes after its last use (made up). The lawyer asks a question every few minutes, so the case file stays warm. The first question after a one-hour lunch is a miss.

What breaks the cache

A common first mistake is a header at the very top of every prompt, such as "Current time: 10:42:07. User: ID 8841." The time changes every second, so the match ends within the first few tokens. The case file is never reused, and the bill does not drop at all.

• A timestamp, session ID or user name placed before the stable parts.
• Adding, removing or reordering tools in the tool list near the top.
• Editing the instructions, even fixing a typo, which breaks the match for every case at once.
• Changing the case file near its start, or sending its pages in a different order.
• Shortening or summarising earlier turns of the chat, which rewrites the middle of the prompt.
• Switching to a different model, because the notes belong to one model.
• Letting the cached start sit unused until the service drops it.

1. 1

   Read the cached token counts

   Check the cached token count on the last 100 calls. Zero on every call means the cache never hit.

2. 2

   Compare two prompts

   Save two prompts sent a minute apart and compare them as text. The first difference is what breaks the match.

3. 3

   Move the changing parts down

   Move the time, user and session details to the end, just before the question.

4. 4

   Check again

   Read the cached token counts on the next day's calls. Most of the prompt tokens should now come from cache.

Prompt caching can cut a large share of the input cost and of the time to first token, but only when a long start is reused many times. It does not reduce the cost or time of writing the answer. Latency means waiting time.

Here is the maths for the 20 questions. Every price is made up to keep the maths easy. Real services publish their own prices, and those change.

Each prompt has a 32,000-token start, a 200-token question and a 500-token answer. Without caching, every question pays Rs 3.22 for 32,200 input tokens and Rs 0.20 for the answer. That is Rs 3.42 per question, or Rs 68.40 for 20.

With caching, the first question is a miss and costs the same Rs 3.42. Each of the other 19 pays Rs 0.64 for the cached start, Rs 0.02 for the question and Rs 0.20 for the answer. That is Rs 0.86 each, or Rs 16.34 together.

The session costs Rs 19.76 instead of Rs 68.40, about 71 percent less. Waiting time follows the same pattern, again with made-up numbers. Time to first token drops from 4 seconds to about 0.5 seconds on a hit.

Writing the 500-token answer still takes 5 seconds (made up) either way. So each full answer takes about 5.5 seconds instead of 9. Caching helps the reading, not the writing.

Real measurements look similar in shape. A January 2026 study of long agent sessions on three hosted services measured cost cuts of 41 to 80 percent. Its results show time to first token improving by only about 6 to 31 percent, though its summary says 13 to 31. Naive full-context caching could even add delay.

Prompt caching saves the model's reading work and still produces a fresh answer every time. Response caching and semantic caching save whole answers and hand them back without calling the model at all.

Response caching. Your app stores each finished answer, keyed by the exact full request. If the identical request comes again, the app returns the stored answer. Example: two juniors in the same office ask the exact same question about the same case file.

Semantic caching. Your app turns each question into an embedding, a list of numbers that captures its meaning. If a new question's numbers are close enough to an old one, the app returns the old answer. Example: "Was notice served in time?" and "Did the notice reach them before the deadline?"

The danger is a false hit: two questions that look alike but need different answers. Research on semantic caches names false hits as a key problem. A question about notice for the first hearing and one about the appeal look almost the same, yet can have opposite answers.

You can stack all three in one app. The diagram shows the order the tool would check them in. Notice that the cheap, exact check comes first, and that the risky semantic path is switched off for legal answers.

Prompt caching does not help when the start of your prompts is short, rarely repeated or keeps changing. It also does nothing for the cost and time of writing long answers.

• One question per document: the first call is always a miss, so you save nothing. If your price list charges extra to write the cache, a one-off prompt can cost more than no caching.
• Short prompts: a hosted service may cache only above a minimum length, which it documents. In vLLM with the default block size, a prompt needs one full 16-token block before its last token.
• Long gaps: if each case is opened once a week, the cached start is likely gone by the next visit.
• Changing content at the top: if each request needs different retrieved paragraphs, put them after the stable part.
• Answer-heavy tasks: if most of the bill is a long drafted petition, caching the input barely moves the total.
• Unshared starts: if every user has a different long document, each one is a miss the first time.

vLLM's own docs make the same point. They say automatic prefix caching shortens only the prefill phase, the reading of the prompt. It does not shorten decoding, the phase that writes new tokens.

vLLM cuts each prompt into fixed-size blocks of tokens and gives every full block a hash, a short fingerprint of its content. A new prompt reuses cached blocks from the start until the first block whose hash is missing.

Blocks and chained hashes

The default block size is 16 tokens, and the --block-size flag changes it. Some model types use other block sizes, such as hybrid models that mix attention with other kinds of layers. Each block's hash covers three things: the previous block's hash, the block's own tokens and extra keys. Extra keys include fine-tuning adapter IDs, image hashes and an optional cache salt.

Because each hash includes the previous one, a block's hash stands for the whole prompt up to its end. That is why one changed token at the top changes every hash after it. vLLM caches only full blocks, so a partly filled last block is always computed.

A collision is two different blocks getting the same fingerprint. Since vLLM 0.11, the default hash is SHA-256, a cryptographic hash that vLLM's docs say addresses collision risks. The docs warn that the faster xxhash option is not cryptographically secure. They add that its collisions could even leak private data on servers shared by many customers.

What stays cached, and for how long

vLLM 0.32.0's prefix cache in GPU memory has no timer. When a request finishes, its blocks join a free queue but keep their cached notes. When new requests need memory, vLLM evicts the least recently used block first, a policy called LRU.

So in vLLM, "how long does the cache last" means how much GPU memory and traffic you have. A quiet server can keep a case file cached until it restarts. A busy one may evict it between two questions.

One detail is in a source comment, not the user docs, as of 19 October 2026. When every prompt token is already cached, vLLM still recomputes the last token to get the next word's scores. This can redo a whole block, so pin your version if you rely on it.

A runnable model of where a match ends

The script below copies vLLM's matching rule in plain Python: 16-token blocks, SHA-256 hashes chained through the parent hash, and an optional salt in the first block. It is not vLLM's code. Token IDs are stand-in numbers, so it runs anywhere with Python 3.

import hashlib
import pickle

BLOCK = 16  # tokens per block, vLLM's default block size


def block_hashes(tokens, salt=None):
    hashes, parent = [], b""
    for start in range(0, len(tokens) - BLOCK + 1, BLOCK):
        extra = (salt,) if start == 0 and salt else ()
        key = (parent, tuple(tokens[start:start + BLOCK]), extra)
        parent = hashlib.sha256(pickle.dumps(key)).digest()
        hashes.append(parent)
    return hashes


def reused_tokens(cache, tokens, salt=None):
    reused = 0
    for h in block_hashes(tokens[:-1], salt):  # the last token is always computed
        if h not in cache:
            break
        reused += BLOCK
    cache.update(block_hashes(tokens, salt))
    return reused


case_file = list(range(32_000))  # stand-in token IDs for instructions + case file
q1 = list(range(900_000, 900_200))  # a 200-token question
q2 = list(range(800_000, 800_200))  # a different question

cache = set()
print("first question: ", reused_tokens(cache, case_file + q1))
print("second question:", reused_tokens(cache, case_file + q2))

with_time = case_file.copy()
with_time[5] = 777_777  # a timestamp near the top
print("time at token 5:", reused_tokens(cache, with_time + q2))

fixed = case_file.copy()
fixed[23_210] = 555_555  # a correction on page 30
print("fix on page 30: ", reused_tokens(cache, fixed + q2))

print("other customer: ", reused_tokens(cache, case_file + q2, salt="team-b"))

Run it with python3 prefix_match.py. On Python 3.13 it prints these five lines, each counting reused tokens:

first question:  0
second question: 32000
time at token 5: 0
fix on page 30:  23200
other customer:  0

What just happened? The first question finds an empty cache, and the second reuses all 32,000 tokens of the shared start. A change at token 5 drops reuse to zero.

A fix at token 23,210 keeps 23,200 tokens, rounded down to a whole block. A different salt gets nothing, even with an identical prompt.

How to see your hit rate

vLLM counts prefix cache lookups and hits in tokens. On its metrics page they appear as vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, and hits divided by queries is your hit rate. Its periodic stats log line also prints "Prefix cache hit rate" as a percentage, covering only about the last 1,000 requests.

To check a running server on its default port, run: curl -s localhost:8000/metrics | grep prefix_cache_. Expect lines such as vllm:prefix_cache_hits_total, then labels and a running count of tokens. Both counters only grow until the server restarts.

For per-request counts, start vLLM with --enable-prompt-tokens-details. Each response's usage then includes prompt_tokens_details.cached_tokens, counted in tokens.

Those per-request numbers feed a daily report like the one below. Each item in calls holds the prompt_tokens and cached_tokens one response reported.

def cache_report(calls):
    prompt_tokens = sum(c["prompt_tokens"] for c in calls)
    cached_tokens = sum(c["cached_tokens"] for c in calls)
    hit_rate = cached_tokens / prompt_tokens if prompt_tokens else 0.0
    healthy = min(hit_rate, 0.5) == 0.5  # True when at least half came from cache
    status = "OK" if healthy else "ALERT: check the top of the prompt"
    print(f"{cached_tokens:,} of {prompt_tokens:,} prompt tokens from cache "
          f"({hit_rate:.0%}): {status}")
    return hit_rate


good_day = [{"prompt_tokens": 32_200, "cached_tokens": 0}] + \
    [{"prompt_tokens": 32_200, "cached_tokens": 32_000}] * 19
bad_day = [{"prompt_tokens": 32_200, "cached_tokens": 0}] * 20

cache_report(good_day)
cache_report(bad_day)

It prints one line per day. On the good day, 608,000 of 644,000 prompt tokens came from cache. On the broken day, none did:

608,000 of 644,000 prompt tokens from cache (94%): OK
0 of 644,000 prompt tokens from cache (0%): ALERT: check the top of the prompt

Yes, if different customers share one cache. A hit returns the first word sooner than a miss, and that gap can be measured. An attacker who guesses a prefix and sees a fast reply learns that someone else recently sent it.

A 2025 audit of 17 hosted services found global cache sharing across users in seven of them. The authors report that at least five providers made changes after disclosure (paper version of July 2025).

vLLM tracks the same risk as CVE-2025-46570, the public ID for this known security flaw. It is a timing side channel in multi-tenant setups, where one server handles many separate customers (tenants). Its mitigation is the cache_salt request field. vLLM mixes the salt into the first block's hash, so only requests with the same salt share cached blocks.

vLLM's docs say to treat the salt as a secret: a long random value, not a user name or account ID. Without a salt, all requests on the server can share cached prefixes. That is fine for one team, and risky for many customers on one server.

The salt is not full isolation, as of 19 October 2026. vLLM's security docs say customers on one server process still share its caches. For strict isolation they advise a separate vLLM instance per customer.

Explain it as reusing the model's work on a shared prompt start. Then show you know how to make it hit, how to measure it and when it is not worth it. Walk through it in this order.

1. 1

   Spot the repeated prefix

   Point out that every request repeats the same instructions and the same 40-page case file, and only the question changes.

2. 2

   Explain the mechanism

   Say the service keeps the KV cache for that start and reuses it only on an exact token match from the first token.

3. 3

   Order the prompt

   Show stable parts first and changing parts last, with timestamps and user details moved to the end.

4. 4

   Do the maths

   Estimate cost and time to first token with and without caching. Say clearly that writing the answer costs the same.

5. 5

   Size the cache

   For a self-hosted server, say cached blocks live until memory runs short, so GPU memory and traffic decide the hit rate.

6. 6

   Measure it

   Log cached tokens per call, track the hit rate and alert when it drops.

7. 7

   Name the limits

   Say when it does not help, how you isolate customers with a per-tenant salt, and why semantic caching is too risky here.