Prompt Caching Explained: Faster, Cheaper LLM Calls

You pay for the model to read the same long text on every call, and the user waits while it reads. Nothing carries over between two calls unless prompt caching keeps it.

Picture a common setup. An online grocery shop runs a support chat assistant that answers questions about orders, refunds and delivery. The setup is made up, but the problem is real.

Every request to the model carries the same three big parts. They are the assistant's instructions, descriptions of the tools it may call, and the shop's 30-page policy handbook. Only the last line really changes: the customer's new message.

In tokens, the shared part is about 30,000 (made up), and a typical message is about 100. So over 99 percent of each prompt is text the model has already read thousands of times today.

The shop handles about 1 lakh (100,000) messages a day. With made-up prices, that is Rs 3.13 (Indian rupees) per message and Rs 3.13 lakh a day, mostly for re-reading the handbook. This guide shows how prompt caching cuts that, and how to lay out the prompt so it actually works.

Six words carry the rest of this guide. Each one gets a plain meaning and an example from the support assistant.

Prompt. The prompt is everything your app sends to the model in one request. Example: the instructions, tool descriptions, handbook, chat so far and the new message, joined in that order.

Token. A token is a small piece of text, roughly a short word or part of a word. Services count prompts and answers in tokens, and hosted ones charge per token. Example: the handbook alone is about 25,000 tokens (made up).

Prefix. A prefix is the beginning part of something. The prefix of a prompt is everything from its first token up to some point. Example: instructions, tools and handbook form a 30,000-token prefix shared by every chat.

KV cache. Before writing a word, the model reads the whole prompt and builds working notes for every token. Engineers call these notes the key-value cache, or KV cache. Prompt caching keeps these notes so a later request can reuse them.

Cache hit and cache miss. A cache is a store of saved work. A hit means the saved notes were found and reused; a miss means the work is done from scratch. Example: a message is a hit if the handbook's notes are still saved.

Time to first token. This is how long the user waits until the first word of the answer appears. Reading the prompt happens before that first word, so it is the wait that prompt caching shortens.

Prompt caching means the model service keeps the working notes it built for the start of a prompt. When a later prompt starts with exactly the same tokens, the service loads those notes instead of rebuilding them.

For the support assistant, the 30,000-token shared part produces the same notes on every request. With caching, the service builds them once and then reuses them. It reads only the new tokens at the end, about 100 for a fresh message.

This does not change what the model computes; it only skips repeated work. vLLM, a popular open-source model server, says its prefix caching won't change model outputs. In our own toy model, built in NumPy, a Python maths library, cached and fully recomputed results differed only in the 16th decimal place. That is rounding noise.

The diagram follows one customer message. Look at the split after the cache check: a hit skips the slow reading step, and a miss reads everything and saves its notes for the next request.

Hosted services differ in the details. A January 2026 study of three hosted services found that some cache automatically, while others ask you to mark the part to cache. It reported minimum prompt lengths of 1,024 to 4,096 tokens.

The same study reported cache lifetimes from 5 minutes to 24 hours, depending on the service. These rules change, so check your provider's current docs rather than hard-coding them from memory.

Because a token's notes, taken across all the model's layers, depend on every token before it. Change one token near the top, and the notes for every later token change too, even if the later text is identical.

The same two-layer toy model shows this. Changing token 5 of a 300-token input left the second-layer results for tokens 0 to 4 untouched. It changed the results for every token from 5 onwards.

Think of two trains leaving Mumbai on the same track. They share every station until the first junction where their routes split. In this picture, they never share track again after that, even if both later pass stations with the same names.

The map: the track is the prompt, each station is a token, and the junction is the first different token. The shared track is what the cache can reuse. Where the picture breaks: real tracks can rejoin, but a prompt never rejoins its cached copy.

A 2025 audit of 17 hosted services notes that several providers state hits occur only for exact prefix matches. Research systems such as Prompt Cache can reuse repeated segments elsewhere in a prompt, but they need extra care with token positions.

Pause and predict. The assistant adds the current time, to the second, as the first line of every prompt. Everything else stays the same. How much of the 30,000-token shared part is reused on the next message?

Almost none. The time changes every second, so the match ends within the first few tokens. Every request reads the whole handbook again, so the bill does not drop. Where writes cost extra, it even rises.

Order the prompt from the most widely shared part to the least shared part. Text shared by every user goes first, text shared within one chat goes next, and text unique to this message goes last.

This is stricter than "stable first, changing last". A customer ID never changes within a chat, but it differs between customers. Put it above the handbook and every customer gets a separate cached copy of the handbook. Put it below and every chat shares one.

The diagram shows the same order as a chain. Follow it from left to right and notice the two places where a cache can stop.

Here is the assistant's prompt builder in plain Python. It joins the layers in the table's order, then compares pairs of prompts to show how much of their start is shared. The texts are short stand-ins, so it runs anywhere with Python 3.

import json
import os

INSTRUCTIONS = "You are the support assistant for an online grocery shop. Be brief."
TOOLS = [
    {"name": "track_order", "description": "Find where an order is now"},
    {"name": "start_refund", "description": "Start a refund for one order"},
]
HANDBOOK = "\n".join(f"Policy {n}: refund and delivery rules." for n in range(1, 301))


def build_prompt(customer_id, history, message, today):
    layers = [
        INSTRUCTIONS,                                 # shared by every chat
        json.dumps(TOOLS, sort_keys=True, indent=2),  # shared by every chat
        HANDBOOK,                                     # shared by every chat
        f"Customer ID: {customer_id}",                # same for one chat
        *history,                                     # grows at the end
        f"Today is {today}. New message: {message}",  # this message only
    ]
    return "\n\n".join(layers)


def shared_start(old, new):
    common = len(os.path.commonprefix([old, new]))
    print(f"{common:,} of {len(new):,} characters shared ({common / len(new):.0%})")
    return common


day = "19 October 2026"
first = build_prompt(5521, [], "Where is my order?", day)
second = build_prompt(5521, ["Where is my order?", "Out for delivery."],
                      "Can I change the address?", day)
other = build_prompt(7310, [], "I want a refund.", day)

shared_start(first, second)  # same chat, next message
shared_start(first, other)   # a different customer
shared_start("Time: 10:42:07\n\n" + first, "Time: 10:42:08\n\n" + second)

Save it as prompt_layers.py and run python3 prompt_layers.py. On Python 3.13 it prints three lines, one for each pair of prompts:

11,856 of 11,959 characters shared (99%)
11,850 of 11,911 characters shared (99%)
13 of 11,975 characters shared (0%)

What just happened? The next message in the same chat shares 99 percent of its characters with the first. A different customer also shares 99 percent, because the customer ID sits below the handbook.

With the time on the first line, only 13 characters match: everything before the last digit of the time. That is the pause-and-predict answer, measured.

The check counts characters, while the cache counts tokens. Some servers, including vLLM, also cache in fixed-size blocks of tokens. So the reused part ends at or a little before the first difference this check finds.

Where retrieved text and tool results go

Retrieval-augmented generation (RAG) adds paragraphs fetched for each question to the prompt. Those paragraphs change per message, so they belong in the last layer, next to the new message.

Placed above the history, they would stop the history from being reused. Tool results, the data a tool sends back mid-chat, join the history at its end, never above it. The January 2026 study gives similar advice: put dynamic content, such as timestamps and user details, at the end of the system prompt.

Read the cached-token count in the usage data each response returns, then divide cached input tokens by all input tokens. That share is your cache hit rate, and it should be high on a long, repeated prompt.

The January 2026 study recorded up to three counts from service responses: normal input tokens, cached tokens read and tokens written to the cache. Field names differ by service, so look them up in your provider's docs.

If your service charges for writes, a high write count with few reads also signals a broken layout. It means the service keeps saving starts that nobody reuses.

1. 1

   Read the numbers

   Pull the cached-token count for the last 100 calls that share the long start. Zero on every call means the cache never hit.

2. 2

   Save two real prompts

   Capture two prompts sent a minute apart, exactly as sent, after templating and JSON encoding. Logging the template is not enough.

3. 3

   Find the first difference

   Run a shared-start check like the one above on the two prompts. Print a few characters around the first difference.

4. 4

   Fix what you found

   Move it below the shared layers, or make it come out the same every time by sorting it, fixing it or removing it.

5. 5

   Check again

   Read the counts on the next day's calls. On the made-up numbers, a hit should show about 30,000 cached tokens out of 30,100.

It cuts the price and time of re-reading the shared start, often by most of it. It never cuts the cost of writing the answer. Where a service charges extra to write the cache, a start used only once costs more.

Here is the maths for the support assistant. Every price is made up to keep the numbers easy. Real services publish their own prices, and they change.

• Normal input tokens: Rs 1 per 10,000 tokens.
• Input tokens read from the cache: Rs 0.10 per 10,000 tokens, a tenth of normal.
• Input tokens written to the cache: Rs 1.25 per 10,000 tokens, on services that charge extra to save notes.
• Output tokens, the answer: Rs 4 per 10,000 tokens, unchanged by caching.
• Each message has the 30,000-token shared start, a 100-token message and a 300-token answer.

Without caching, a message pays the normal price for all 30,100 input tokens, plus the answer. A hit pays the cached price for the 30,000-token start instead. A miss that saves the start pays the write price for it, so it costs more than no caching.

Why the layer order matters for the bill

Now the layer order shows its value. Assume 12,500 chats of 8 messages each, which is the shop's 1 lakh messages. To keep the maths short, ignore the chat history.

With the customer ID at the top, each chat gets its own cache: one miss and seven hits. With the ID below the handbook, all chats share one cached start, and a made-up 99 percent of messages hit. The table puts every case side by side.

The prices are the same in the last two rows; only the layer order changed. Moving one line below the handbook lifts the daily saving from 72.5 to about 85 percent.

When does a cached start pay for itself? Used once, it costs 1.25 times one normal read of it. Used twice, it costs 1.35 times one normal read instead of 2. So on these prices it pays back on the first reuse, if that reuse comes before the cache expires.

What happens to waiting time

Waiting time improves in the reading step only. With made-up timings, reading 30,000 tokens takes 3 seconds on a miss and 0.3 seconds on a hit. Writing a 300-token answer takes 3 seconds either way, so a reply takes 3.3 seconds instead of 6.

Measured gains can be smaller than toy maths suggests. The January 2026 study ran long agent sessions, where the model calls tools many times in a row, on three hosted services. It measured cost cuts of 41 to 80 percent but time-to-first-token gains of only about 6 to 31 percent. Its summary says 13 to 31, but its own results table goes as low as 6.1.

It does not help when the shared start is short, rarely repeated or keeps changing. It also does nothing for the cost and time of writing long answers.

• One-off prompts: a long document used once is always a miss, and costs more where writes are charged extra.
• Short prompts: hosted services usually set a minimum length, and the study above found minimums of 1,024 to 4,096 tokens.
• Long gaps: a cached start is dropped after its lifetime, or earlier when the service needs the memory.
• Answer-heavy work: if most of the bill is long drafted replies, caching the input barely moves the total.
• Unshared documents: if every customer uploads a different long file, each one is a miss the first time.
• Content that must sit at the top and changes often: no ordering trick saves a prefix that is never repeated.

Anything that makes the same logical prompt render as different text breaks the match. Three habits cause most of it: unsorted JSON, Python sets and stamps inside files you think are stable.

JSON keys in a different order

Python's json.dumps writes keys in the order the dictionary holds them, unless you pass sort_keys=True. Two equal dictionaries built in different orders give different text, and so different tokens.

Our test built the same tool description twice with keys in a different order. Python said the dictionaries were equal, but the JSON text differed. With sort_keys=True, the two texts matched exactly.

Indentation and separators change the text too. Pick one json.dumps call for every prompt layer and reuse it everywhere.

Python sets that change order between processes

By default, Python seeds string hashing with a random value in every new process. So a set of strings can list its items in a different order in each server process.

Build a tool list from a set, and two app server processes sharing the same traffic send two different prompts. Each process then keeps its own cached copy of the start, and every restart can bring a new order and a fresh miss. This short shell loop shows the effect by fixing three different seeds:

for seed in 1 2 3; do
  PYTHONHASHSEED=$seed python3 -c "print(list({'track_order', 'start_refund', 'find_store'}))"
done

On Python 3.13.16 it printed three different orders:

['start_refund', 'track_order', 'find_store']
['track_order', 'start_refund', 'find_store']
['find_store', 'start_refund', 'track_order']

Replace list with sorted in the same loop, and all three lines become identical:

['find_store', 'start_refund', 'track_order']
['find_store', 'start_refund', 'track_order']
['find_store', 'start_refund', 'track_order']

PYTHONHASHSEED only makes the demo repeatable. In real code, sort the list rather than pinning the seed.

Stamps inside stable text

Generated files often carry a "generated at" line, a request ID or a build number. If that line sits inside the instructions or the handbook, every rebuild starts a new cache. Strip it, or move it to the last layer.

Trim in large steps, not one turn at a time. Each trim rewrites the history layer from its first line, so every trim costs one full re-read of the history.

A simple trimming rule keeps the last few turns and drops the oldest one whenever a new turn arrives. Then the history's first line changes on every request, so nothing after the customer details is reused.

We modelled this with exact prefix matching. The model used the 30,000-token shared start, 400-token turns and a made-up 4,000-token history limit, over 40 turns. It assumed nothing expired from the cache.

Dropping one turn each time re-read 110,700 history tokens. Dropping the oldest half whenever the limit was hit re-read 21,900, about a fifth as many.

The trade-off is context. Halving keeps fewer old turns right after a trim, so check that answers stay good. Summarising old turns rewrites the history too, so do it rarely and in one go.

vLLM calls it automatic prefix caching. As of 19 October 2026, vLLM 0.32.0 turns it on by default for models that support it. It caches prompts in blocks of 16 tokens by default.

vLLM's feature page still says to set enable_prefix_caching=True to turn it on. In the 0.32.0 source it is already on, unless the model or hardware does not support it. To switch it off, start the server with --no-enable-prefix-caching.

Each block is fingerprinted together with everything before it, and only full blocks are cached. So with default settings, a cached start ends on a block boundary. One change inside a block loses that block and every block after it.

vLLM's built-in prefix cache has no timer. Cached blocks stay until the server needs the memory, and then the least recently used blocks go first. On a self-hosted server, spare memory and traffic decide how long a start stays cached.

To see per-request numbers, start the server with --enable-prompt-tokens-details. Each response's usage then includes prompt_tokens_details.cached_tokens, counted in tokens. A separate SevyDevy guide covers vLLM prefix caching and cache_salt, a per-customer value that stops customers sharing cached blocks, in depth.

It can, if one cache is shared across customers. A hit returns the first word sooner than a miss, so someone who times replies can guess whether another user recently sent a given start.

A 2025 audit of 17 hosted services found cache sharing across users at seven of them. On vLLM, a secret cache_salt value per customer stops different customers sharing cached blocks. vLLM's docs add that the salt is not full isolation between tenants.

In the layered design, only the instructions, tools and handbook should ever be shared across customers. Customer details and history sit below them, so they stay specific to one chat.

Explain it as reusing the model's reading work on an identical prompt start. Then show how you would lay out the prompt, check it and know when it is not worth it. Walk through it in this order.

1. 1

   Find the repeated start

   Name the shared parts, such as instructions, tools and a handbook, and their size in tokens.

2. 2

   State the rule

   Say the service reuses notes only for an exact token match from the first token, so one change ends reuse.

3. 3

   Layer the prompt

   Order parts by who shares them: every user first, then one chat, then one message.

4. 4

   Do the maths

   Estimate cost and time to first token with and without caching, including any extra charge for writes.

5. 5

   Guard it

   Add a rendering test for the shared layers and an alert on the cache hit rate.

6. 6

   Name the limits

   Mention one-off prompts, long answers and how shared caches are isolated between customers.