Semantic Caching for LLM Apps: Reuse Answers Without Wrong Hits

It pays for the same answer again and again. People ask the same few questions in many different words, and each new wording looks like a new question to the bot.

Picture a common setup. An electricity supplier runs a help bot on its website and app, and the bot answers about 20,000 questions a day. The setup is made up, but the problem is real.

Behind the bot sits a large language model, which engineers usually just call the model. For each question, the bot sends the model a prompt: its instructions plus the customer's words. Each answer costs about Rs 0.40 and takes about 3 seconds (made up).

The customer service team reads the chat logs every Monday. One week they circle five messages from the same morning:

• "How do I pay my light bill online?"
• "Can I pay electricity bill from my phone?"
• "online payment for bijli bill"
• "Which app should I use to pay my bill?"
• "pay bill online steps please"

All five need the same answer: open the app, enter the consumer number, and pay by UPI or card. Yet the bot called the model five times and paid five times. Across 20,000 questions, that adds up to Rs 8,000 a day.

The team first tries a normal cache, which saves an answer under the exact text of the question. It catches only about 6 percent of questions (made up), because people rarely type the same words twice. What the team wants is a cache that notices when two questions mean the same thing, and that idea is called semantic caching.

Think of a senior clerk at the supplier's payment counter. She keeps a notebook of standard replies, and when anyone asks about paying online, in any words, she reads out the same reply.

Where the picture breaks: the clerk understands what people mean, and she hears when rates change. A semantic cache only compares lists of numbers, and it learns about a change only when your code tells it.

You need six terms: embedding, similarity score, threshold, cache hit, false hit and expiry. Each one comes back in every later section.

Embedding. An embedding is a list of numbers that stands for the meaning of a piece of text. A separate model, called an embedding model, makes it, and it is trained so that texts with similar meanings get lists that sit close together.

Similarity score. A similarity score says how close two embeddings are. A common measure is cosine similarity, which compares the directions of two lists: 1 means the same direction, and the lowest possible score is -1.

Threshold. The threshold is the lowest similarity score you accept as the same question. If you set it to 0.92, a match scoring 0.94 counts as the same question and one scoring 0.85 does not.

Cache hit and cache miss. A hit means a saved answer was close enough and is reused. A miss means nothing was close enough, so the bot asks the model and may save the new answer.

False hit. A false hit, or wrong hit, is a cache hit that returns the answer to a different question. Someone asks how to pay a bill late and gets the normal payment steps without the late fee.

Expiry. Expiry is the time after which a saved answer is thrown away. Engineers also call it time to live, or TTL. In this guide, general answers live for 7 days.

Semantic caching means saving a model's whole answer and reusing it for any later question with the same meaning. It compares embeddings instead of exact words, so many wordings can share one saved answer.

Each question goes through the same steps. The bot turns it into an embedding, keeps only saved entries whose exact labels match, finds the closest one and checks its score. On a miss, it asks the model and saves the answer only if it is safe to share.

Follow the diagram from the customer's question. Notice that one decision box in the middle decides whether the model is called at all.

Three rules carry most of the safety. The search looks only at entries with the same key, a set of exact labels such as language and customer type. A safety check runs before anything is saved, so an answer built from private data never enters the cache.

The third rule is expiry. Every saved answer gets a time limit, so a stale answer cannot live forever even if nobody remembers to clear it.

You choose it by testing several thresholds on real pairs of questions and counting both hits and wrong hits. No single number suits every app, because scores depend on the embedding model and on how alike your questions are.

A lower threshold raises the hit rate, the share of questions answered from the cache, and also the number of wrong answers. A higher threshold gives fewer wrong answers and fewer hits, so you save less. The vCache research paper (2025) notes that existing semantic caches use one fixed threshold for every request.

So the team builds a test set. They take 500 pairs of real questions from the logs, and the service team marks each pair as same answer or different answer. Then they try four thresholds on the pairs and on a week of logs.

Small changes in words can hide big changes in meaning. These pairs share almost every word and still need different answers:

• "How do I pay my bill?" and "How do I pay my bill late?" Only the second involves a late fee.
• "New connection for my shop" and "New connection for my home" need different forms and fees.

Pause and predict: if a wrong pair scores higher than a right pair, which threshold keeps the right one and drops the wrong one? None does. Any threshold low enough to accept the right pair also accepts the wrong one.

The runnable cache later in this guide shows exactly this. With its stand-in embedding, the late-payment question scores 0.894 against the saved payment question, while an honest rewording scores only 0.781.

So the threshold cannot be your only defence. Exact key labels and a clear rule about what may be cached do the work that no threshold can.

Never put answers that depend on who is asking in a shared cache, and never cache answers that change quickly or come from taking an action. A shared entry stays fixed, so it is only safe when the same answer is right for everyone who asks.

Go back to the clerk. Her notebook holds the steps to pay a bill, but never one customer's bill amount, because the next person in the queue is someone else.

• Personal answers: bill amounts, meter readings, payment history, the address on an account. Reusing one would show a customer someone else's data.
• Live answers: power cuts, repair times, today's queue at the service centre. These go wrong within hours.
• Action answers: a new complaint number or a booked meter check. Every customer needs their own.
• Follow-up questions such as "And for a shop?", unless you cache them together with their context, as explained later.

Leaking personal data is a listed risk for LLM apps. OWASP, a non-profit group that publishes security guidance, ranks sensitive information disclosure second in its 2025 list for LLM applications. One of its example scenarios is a user receiving another user's personal data.

A shared cache that saved one customer's bill amount would cause exactly that. So the bot's safety check asks two questions about every new answer before saving it. Start at the top of the chart and follow the answers.

In practice, the bot already knows which data each answer used. If the answer read the customer's account or the live outage feed, or booked something, the check refuses to save it. Because the rule reads the list of data sources, not the text, a cleverly worded question cannot talk its way past it.

This rule only stops personal answers from being saved. A personal question can still match a saved general answer and get a reply that ignores the customer's account. If your bot picks its data sources before answering, skip the cache lookup for questions that need account or live data.

Beginners can stop here. You now know what a semantic cache does, how the threshold trades savings for wrong answers, and what must never go into it. The rest of the guide is for engineers who will build one.

Put every fact that changes the right answer into the cache key, such as language, customer type and version labels. A cache key is a set of exact labels an entry must match before its meaning is even compared. Add the user id only for a private cache meant for one person.

The key filter runs first, as an exact match. A shop owner's question is only compared with answers saved for shops. So "New connection for my shop" can never match the home answer, however high the score.

Here is the key the bot builds. At the start of each chat, the bot asks the customer to pick home or shop. The rules label changes whenever the supplier's tariff, its list of electricity rates, or its rules change. The prompt label changes whenever someone edits the bot's prompt.

RULES_VERSION = "tariff-2026-10"
PROMPT_VERSION = "prompt-v14"


def make_key(request, scope="shared"):
    key = (
        request.language,        # "en" or "mr" (Marathi)
        request.customer_type,   # "home" or "shop"
        RULES_VERSION,           # new tariff, new key
        PROMPT_VERSION,          # new prompt, new key
    )
    if scope == "private":
        key += (request.account_id,)   # only this customer
    return key

For an English home customer, make_key returns ('en', 'home', 'tariff-2026-10', 'prompt-v14'). The private scope adds the account id as a fifth label, so nobody else can ever match that entry.

The language label means a Marathi question never gets an English answer. The rules and prompt labels mean a new tariff or a new prompt starts on fresh keys. Old answers are simply never found again, and their expiry clears them out later.

Web caching draws the same line between shared and private. The HTTP caching standard, RFC 9111, separates a shared cache used by many users from a private cache dedicated to one. A response marked private must not be stored by a shared cache.

What about follow-up questions?

A follow-up question only makes sense with the messages before it. Say a customer first asks about the fee for a new home connection. Next they type "And for a shop?" and nothing else. On its own, that question means almost nothing, so its embedding could match the wrong entry.

There are two safe options. Skip the cache for follow-ups, or first rewrite the follow-up into one full question, such as new connection fee for a shop, and embed that.

MeanCache, a 2024 research system, takes a third route. It stores a context chain, the earlier questions in the chat, with every cached query. A similar follow-up counts as a hit only when its own earlier questions also match that chain.

Cached answers should expire on a timer, and also the moment the facts behind them change. The timer catches changes nobody told you about. Clearing entries when you know a fact changed is called invalidation.

The bot uses both. Payment steps and document lists live for 7 days. Answers about rates carry the rules label in their key. On the day new rates are announced, the team changes the label, and every old rate answer becomes unreachable at once.

Here is the full setup. Look for the two ways stale answers leave: the expiry timer inside the cache, and the update job that runs when the tariff store changes.

Two more triggers are easy to forget. If someone edits the prompt so answers become shorter, old answers no longer match the new style. That is why the prompt label sits in the key.

A new embedding model is the other trigger. Its numbers usually cannot be compared with the old model's numbers unless its maker says the two were built to match, and the lists often have a different length. Clear the cache, or embed every saved question again with the new model.

You can see this with two stand-in models in our test. They made lists of 256 and 384 numbers. Comparing just the first 256 numbers of each, the same sentence scored only 0.089 against itself, where 1 would mean identical.

You can build the whole idea in about 70 lines of plain Python and watch it make a false hit. The code below needs no installs; we ran it on Python 3.13.16.

One honest shortcut: a real embedding model needs a download, so the embed function here is a stand-in. It counts three-letter pieces of the text, which means it sees spelling, not meaning. The key filter, threshold, expiry and safety check work as they would with a real model.

import math
import zlib

THRESHOLD = 0.85                 # suits this stand-in only; test your own model
TTL_SECONDS = 7 * 24 * 3600      # expiry: 7 days
NEVER_CACHE = {"account", "outage_feed", "booking"}


def embed(text, dims=256):
    # Stand-in for an embedding model: counts 3-letter pieces of the text.
    # It sees spelling only; a real model also matches different words.
    t = " " + " ".join(text.lower().strip(" ?.!").split()) + " "
    v = [0.0] * dims
    for i in range(len(t) - 2):
        v[zlib.crc32(t[i:i + 3].encode()) % dims] += 1
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]


def similarity(a, b):            # cosine similarity of two unit-length lists
    return sum(x * y for x, y in zip(a, b))


class SemanticCache:
    def __init__(self):
        self.entries = []

    def lookup(self, question, key, now):
        vec = embed(question)
        best, best_score = None, None
        for e in self.entries:
            if e["key"] != key or e["expires"] <= now:   # exact filters first
                continue
            s = similarity(vec, e["vector"])
            if best_score is None or s > best_score:
                best, best_score = e, s
        if best_score is not None and best_score >= THRESHOLD:
            return best, best_score                      # cache hit
        return None, best_score                          # cache miss

    def save(self, question, answer, key, sources, now):
        if sources & NEVER_CACHE:                        # decide from data used
            return False
        self.entries.append({"question": question, "vector": embed(question),
                             "answer": answer, "key": key,
                             "expires": now + TTL_SECONDS})
        return True


cache = SemanticCache()
now = 0                          # seconds; a fixed clock keeps the demo repeatable
home = ("en", "home", "tariff-2026-10", "prompt-v14")
shop = ("en", "shop", "tariff-2026-10", "prompt-v14")

print(cache.save("How do I pay my bill?",
                 "Open the app, enter your consumer number, pay by UPI or card.",
                 home, {"faq"}, now))
print(cache.save("What is my bill amount this month?",
                 "Your bill is Rs 1,240.", home, {"account"}, now))


def ask(question, key, t):
    hit, score = cache.lookup(question, key, t)
    shown = "no entry" if score is None else f"{score:.3f}"
    print(f"{question:40} {shown:>8}  {'HIT' if hit else 'MISS'}")


ask("how do i pay my bill", home, now)
ask("How can I pay my bill?", home, now)
ask("Can I pay the light bill from my phone?", home, now)
ask("How do I pay my bill late?", home, now)
ask("How do I pay my bill?", shop, now)
ask("How do I pay my bill?", home, now + 8 * 24 * 3600)

Save it as cache_demo.py and run python3 cache_demo.py. The middle column is the best similarity score among entries with the same key. You should see exactly this:

True
False
how do i pay my bill                        1.000  HIT
How can I pay my bill?                      0.781  MISS
Can I pay the light bill from my phone?     0.424  MISS
How do I pay my bill late?                  0.894  HIT
How do I pay my bill?                    no entry  MISS
How do I pay my bill?                    no entry  MISS

What just happened? The first save printed True, and the bill-amount save printed False, because that answer read account data. The lower-case repeat scored 1.000 and hit.

The next lines are the lesson. The honest rewording scored 0.781 and missed. The late-payment question scored 0.894 and got the normal payment steps, which is a false hit.

No threshold fixes that pair, because the wrong question scores higher than the right one. A real embedding model is built to place rewordings like the light-bill question much closer than this stand-in does. Pairs that share almost every word, like the late-payment one, are what your test set must include.

The last two lines show the key and the expiry at work. The shop key has no entry at all, and 8 days later the 7-day entry has expired.

Because many vector databases, stores built to find the nearest embeddings fast, return a distance, not a similarity. With a distance, smaller means closer, so a hit means the distance is at most a limit, not at least a threshold.

Redis is one example. Its vector search docs, as of 19 October 2026, define the COSINE metric as cosine distance: 1 minus the cosine similarity. They also say smaller values mean closer vectors, and that cosine distance is bounded by 2.

So a similarity threshold of 0.92 becomes a distance limit of 0.08. Redis range queries take such a limit as a radius, the largest distance allowed.

Two small traps follow. Code that keeps the old greater-or-equal test now rejects close matches and accepts far ones. And in Python, 1 - 0.92 prints 0.07999999999999996, so write 0.08 directly or round it.

Before you copy a threshold from a blog or a library default, check whether your store returns a similarity or a distance. The same number means opposite things in the two.

Semantic caching saves roughly the share of questions it answers from the cache, minus the cost of embedding every question and of cleaning up wrong hits. Do this maths before you build anything; the numbers below belong to the made-up setup.

The bot gets 20,000 questions a day. A model answer costs Rs 0.40 and takes 3 seconds. Embedding plus a cache search costs Rs 0.01 and takes 0.1 seconds. At a threshold of 0.92, 35 percent of questions are hits.

Without a cache, the bill is 20,000 times Rs 0.40, so Rs 8,000 a day. With the cache, every question is embedded first: 20,000 times Rs 0.01, so Rs 200. The 13,000 misses still go to the model, which costs Rs 5,200, so the total is Rs 5,400 and the saving is Rs 2,600 a day.

Wait time improves too, but less than you might hope. Hits take 0.1 seconds, and misses now take 3.1 seconds because they pay for the search first. The average comes to about 2.05 seconds, down from 3.

Now count the wrong hits. There are 7,000 hits a day, and 1 in 100 is wrong, so 70 customers get a wrong answer. Say each one phones the helpline, and each call costs the supplier Rs 15, which is Rs 1,050 a day.

So the real saving is about Rs 1,550 a day, not Rs 2,600. With the same 7,000 hits but 3 wrong in 100, the calls would cost Rs 3,150, and the cache would lose Rs 550 a day. Measuring false hits matters as much as the hit rate.

You measure false hits by checking a random sample of cache hits regularly and counting how many returned the wrong answer. The false hit rate is the number of wrong hits divided by the number of hits you checked.

This needs the hit log from earlier, with both questions and the score. Without it you know how many hits you had, but not how many were right.

1. 1

   Sample every day

   Pick 200 random hits from yesterday's log. Random matters, so you do not only check the easy ones.

2. 2

   Have people mark them

   The service team reads each pair and marks it right or wrong. Say they find 2 wrong out of 200, which is 1 in 100. One day is noisy: if you act only when more than 4 of 200 are wrong, a true 3 in 100 rate slips past about 28 percent of the time. Add up a week, 1,400 checks, before you move the threshold.

3. 3

   Look at the scores of wrong hits

   If most wrong hits score just above the threshold, it is too low for some kinds of question. Check whether a missing key label would have stopped them.

4. 4

   Watch customer signals

   Count how often a customer asks again, or asks for a person, within a minute of a cached answer. A sudden jump is an early warning.

5. 5

   Rerun the test set on every change

   Run the 500 labelled pairs again whenever you change the embedding model, the threshold or the key. A new embedding model gives different scores, so the best threshold may move.

Another option is a model as a judge: a second model call that asks whether a saved answer fits a new question. Running it on every hit would add a model call to every hit and eat into the savings.

The 2026 tiered-cache paper runs its judge in the background instead, when the closest vetted answer scores just below the threshold. Matches the judge approves are added to the dynamic tier for next time.

Another idea from research is to drop the single threshold. The vCache paper learns a separate threshold for each cached prompt, and its authors report that it stays within an error rate the user sets. You do not need this on day one.

Explain it as a shortcut in front of the model, then spend most of your time on what can go wrong. Interviewers want to hear about wrong hits, private data and stale answers, not just the hit rate.

1. 1

   Check the traffic

   Ask how many questions repeat in meaning. A help bot fits well. A tool where every question is unique does not.

2. 2

   Split what can be cached

   Say that personal, live and action answers skip the cache, decided from the data each answer used.

3. 3

   Design the key

   Use exact filters first: language, customer type, rules version and prompt version. Then search by meaning among the entries that match.

4. 4

   Pick the threshold with data

   Describe a labelled test set from real logs, and the trade-off between more hits and more wrong hits. Say whether your store returns similarity or distance.

5. 5

   Plan expiry and invalidation

   Give every entry a timer, and change the version label when the facts behind an answer change.

6. 6

   Do the maths

   Show the saving minus the embedding cost minus the cost of wrong hits.

7. 7

   Measure false hits

   Mention daily samples, customer signals and a test set you rerun after every change.