Query Rewriting for RAG: Multi-Query, Decomposition and HyDE

The search misses articles you already have, because people write the way they talk and help articles are written the way companies write. The bot then answers from the wrong pages, even though the right one exists. Query rewriting for RAG fixes this by turning the message into a clearer search query before the search runs.

Picture a common setup. An electronics retailer in India sells phones, speakers, earbuds and TVs online. Its support chat bot answers from about 1,200 help articles. The setup is made up, but the problem is real.

A customer tells the bot she bought the SP-200 speaker last month. Her next message reads: "it stopped charging after the rain. covered?"

The naive version, which the bot ran, sends that message straight to search, word for word. The top results were articles about charging cables and battery tips. The article she needed, on water damage and the 1-year warranty, never came up.

The support lead pulled 200 chats where customers gave up and asked for a person. In 140 of them, the right help article existed. The search itself worked. The questions were the problem.

Sorted into groups, the 140 misses look like this. Each group gets its own fix later in the guide.

• Missing context. A follow-up like the speaker one only makes sense with the earlier messages, which the search never sees. Fix: a chat-history rewrite.
• Different words. Customers write money back, while the article is called Refund policy. Fix: multi-query search.
• Noise and spelling. Greetings, order numbers, typos such as warenty, and Hinglish (Hindi typed in English letters) such as charge nahi ho raha hide the one line that matters. Fix: cleanup, then a rewrite.
• Several questions in one. EMI (paying in equal monthly instalments), wall mounting and delivery date arrive in one message. Fix: decomposition.
• Too specific. The question names one exact case, but the answer lives in a general policy. Fix: a step-back question.

Eight words carry the rest of this guide. Each is explained once here and used the same way after.

• RAG: retrieval-augmented generation. The app searches your documents first, then a model writes the answer from what was found.
• Model: a large language model, the AI program that reads and writes text. A small model is a cheaper, faster one.
• Chunk: a small piece of a help article, a few paragraphs long. The retailer's 1,200 articles make about 9,000 chunks.
• Retrieval: the search step that picks the best chunks for each question, here the top 5.
• Prompt: the text your app sends to the model, including the chunks and the customer's question.
• Embedding: a list of numbers that stands for a text's meaning. Vector search finds the chunks whose numbers sit closest to the question's numbers.
• Keyword search: search that matches the exact words typed, such as an order number.
• Token: a small piece of text, roughly a word or part of one. Model services usually charge by tokens.

Query rewriting for RAG changes the user's message into one or more better search queries before retrieval runs. The step that does it is the rewriter: sometimes plain code, often one short call to a small model.

A 2023 research paper named this pattern Rewrite-Retrieve-Read. The usual order is retrieve, then read, where read means the model reads the chunks and answers. The paper argues there is always some gap between the input text and the knowledge a search needs.

That paper went further than writing a prompt. It trained a small model as the rewriter with reinforcement learning, which means learning by trial and reward. The reward came from the answering model's feedback, and the retriever was a web search engine.

Think of the job card at a two-wheeler service centre. A customer says the scooter makes a funny noise. The service advisor writes the card: check front brake, check chain tension. The mechanic works from the card, not the vague complaint.

Map it part by part. The complaint is the customer's message, the advisor is the rewriter, the card is the query, and the mechanic is the search. Where the picture breaks: an advisor understands scooters, while a rewriter predicts likely words and can confidently write the wrong card.

In the diagram, follow the arrows from the customer's message to the answer. The search box is vector search over the 9,000 chunks.

Clean the text with plain code first, then ask a small model to turn a follow-up into a stand-alone question. These are the cheapest fixes, so try them before anything else.

Cleaning the text with plain code

Query cleanup means small, safe fixes that need no model at all. Every rule is plain code, so you can test it. This cleanup function needs no installs; it was tested on Python 3.11, 3.12 and 3.13.

import re

SPELLING = {"warenty": "warranty", "pwr bank": "power bank", "money back": "refund"}
FILLER = ["please help", "hello", "hi"]

def clean_text(message):
    text = message.lower()
    order_ids = re.findall(r"\bod\d{5,}\b", text)    # order numbers like OD48213
    text = re.sub(r"\bod\d{5,}\b", " ", text)
    for wrong, right in SPELLING.items():
        text = text.replace(wrong, right)
    for word in FILLER:
        text = re.sub(rf"\b{word}\b", " ", text)
    text = re.sub(r"([!?.,])\1+", r"\1", text)       # "??" becomes "?"
    text = re.sub(r"\s+", " ", text).strip(" ,!?.")
    return text, [o.upper() for o in order_ids]

print(clean_text("Hi!! pwr bank from order OD48213 stopped charging, warenty??"))
print(clean_text("hello, money back for opened earbuds?"))

# Output:
# ('power bank from order stopped charging, warranty', ['OD48213'])
# ('refund for opened earbuds', [])

What just happened? The order number moved into its own list, so you can match it exactly with keyword search. The slang and the typo became the words the articles use, and the greeting and doubled punctuation went.

Rules like these can bite. The hi rule above also turns hi-fi speaker into -fi speaker. Keep a file of real messages and rerun the cleaner on it after every rule change.

Rewriting a follow-up with the chat history

A follow-up leans on earlier messages. In the speaker chat, the word it means the SP-200 speaker, but only the chat knows that. The search sees one short line.

The fix is a conversational rewrite. You send a small model the last few messages and the new question, and ask for a stand-alone version. A good rewrite names the product and the problem.

In the code below, ask_model is the function you write to call your model provider. The example passes in a stand-in that returns a fixed reply, so it runs offline.

def rewrite_query(history, question, ask_model):
    if not history:                     # first message: nothing to resolve
        return question
    recent = "\n".join(history[-6:])    # only the last six messages
    prompt = ("Rewrite the last question so it makes sense on its own. "
              "Keep product names and numbers. Do not answer it.\n\n"
              f"Chat so far:\n{recent}\n\nLast question: {question}")
    rewritten = ask_model(prompt).strip()
    return rewritten or question        # empty reply: keep the original

# Stand-in for your model call, so the example runs offline.
def fake_model(prompt):
    return "SP-200 speaker stopped charging after rain, is water damage covered by warranty"

history = ["Customer: I bought the SP-200 speaker last month."]
print(rewrite_query([], "is the SP-200 waterproof", fake_model))
print(rewrite_query(history, "it stopped charging after the rain. covered?", fake_model))
print(rewrite_query(history, "covered?", lambda prompt: "  "))

# Output:
# is the SP-200 waterproof
# SP-200 speaker stopped charging after rain, is water damage covered by warranty
# covered?

What just happened? The first call has no history, so it skips the model and returns the question unchanged. The second sends at most the last six messages, because every extra line costs tokens. Six is a starting point to test, not a rule.

The third call shows the fallback: an empty reply keeps the original question. The prompt also says Do not answer it, because a rewriter that answers puts its own guess into the search.

Multi-query retrieval asks a model for a few different wordings of the question, searches with each one and merges the result lists into one. Different wordings find different chunks, so together they miss less.

This fixes the different-words problem. Take the question can I return opened earbuds. The rewriter adds three wordings: refund for used audio products, earbuds hygiene return rules, and return policy for opened accessories. The original stays in, so four searches run.

How reciprocal rank fusion merges the lists

You merge them with reciprocal rank fusion, or RRF, from a 2009 research paper. Each chunk earns points from every list it appears in, and higher places earn more points.

The points from one list are 1 divided by (k + rank), where rank 1 is the top result. Here k is a fixed number that softens the effect of a top spot. The paper fixed k at 60 after a pilot study and found the exact choice was not critical.

RRF uses only positions, not raw scores. So it can merge lists whose scores cannot be compared, such as a vector search list and a keyword search list.

A small worked example shows the effect. A chunk ranked 1st in one list and missing from the others scores 1/61, about 0.016. A chunk ranked 3rd in three lists scores 3/63, about 0.048. Steady agreement beats one top spot.

In the diagram, one question spreads into four queries. If the four searches run in parallel, the whole step takes about one model call plus the slowest search, not four searches in a row.

Query decomposition splits a question that asks several things into smaller sub-questions. Step-back prompting adds a general version of a very specific question, so the search also finds the rule behind it.

Splitting one question into sub-questions

Query decomposition breaks one question into smaller ones that can each be searched alone. Say a customer asks, in one long message, about EMI on two TVs and whether both get free wall mounting in Patna.

No single help article answers all of that. One search for the whole message returns half-matches, each covering one part badly. So the rewriter writes one sub-question per fact she needs:

• EMI options for the 55-inch TV
• EMI options for the 50-inch TV
• Free wall mounting for TVs delivered in Patna

The bot searches each sub-question, keeps the top 2 or 3 chunks from each, and the model writes one answer covering every part. The idea is close to least-to-most prompting, from a 2022 paper, which splits a hard problem into simpler ones and solves them in order.

Order matters for search too. Independent sub-questions, like the three above, can run at the same time. Now take a customer asking which speaker under Rs 3,000 is cheapest, and whether it is waterproof. The second search needs the first answer, so the wait grows.

Adding a step-back question

A step-back question asks for the general rule behind a specific case. It comes from a 2023 research paper on step-back prompting, where the model first asks about the higher-level concept, then reasons about the original question.

Here is an example from the retailer. A customer opened a power bank 9 days ago, and now it heats up while charging. No article mentions that exact case, so the step-back question is: return policy for opened electronics with a fault.

In the paper's knowledge-heavy question tests, retrieval ran for both the original and the step-back question, and both sets of passages went into the final prompt. You can do the same, or merge the two lists with RRF.

Each query has its own job. The original finds anything specific, such as a known heating fault in that power bank. The step-back question finds the policy.

HyDE, short for hypothetical document embeddings, asks a model to write a fake answer and then searches with that fake answer instead of the question. An answer looks much more like a help article than a question does.

The method comes from a 2022 research paper by Gao and others. Hypothetical means imagined: the model writes what a good answer might look like, without seeing your documents. That text becomes an embedding, and vector search finds the real chunks closest to it.

Back to the speaker. The rewritten question is one short line, while the warranty article is long. HyDE asks the model for a short help article that answers it. A typical fake article talks about liquid damage, warranty exclusions and service centres, like the real one.

In the diagram, notice that the fake answer never reaches the customer. It only steers the search.

The paper is open about the risk: the fake document may contain false details. It argues that the encoder, the model that turns text into an embedding, keeps the pattern of a relevant answer and filters those details out.

Two details matter if you build it. The paper samples several fake documents and averages their embeddings, and it also considers adding the question's own embedding to that average. Its retrieval tests used vector search, with keyword search only as a baseline to beat.

HyDE has limits. If the model knows little about your topic, its fake answer can steer the search the wrong way. It is often the slowest technique here, because a paragraph takes more output tokens to write than a short query.

You add 1/(k + rank) for every list a chunk appears in, then sort by the total. The function below does exactly that. It needs no installs and was tested on Python 3.11, 3.12 and 3.13.

The four lists are made-up search results for the earbuds question, best first, one per wording. Pause and predict: earbud-care tops two lists, while opened-items is only third or fourth in all four. Which one wins?

def rrf(ranked_lists, k=60):
    scores = {}
    for results in ranked_lists:
        for rank, chunk_id in enumerate(results, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

lists = [
    ["earbud-care", "battery-tips", "opened-items", "charging-port"],         # original
    ["refund-timeline", "accessory-returns", "opened-items", "battery-tips"], # refund
    ["earbud-care", "hygiene-rules", "opened-items", "refund-timeline"],      # hygiene
    ["accessory-returns", "hygiene-rules", "refund-timeline", "opened-items"],# policy
]
for chunk_id, score in rrf(lists)[:3]:
    print(f"{chunk_id:18} {score:.4f}")

# Output:
# opened-items       0.0632
# refund-timeline    0.0479
# earbud-care        0.0328

What just happened? opened-items never ranked first, yet it wins with 0.0632, because all four lists found it. earbud-care topped two lists but scores only 0.0328, because the other two missed it.

Now change one number. Call rrf(lists, k=1) and earbud-care jumps to first with 1.0, while opened-items drops to a tie at 0.95. A small k makes a top spot count far more, so k decides whether agreement or one strong list wins.

So check k on your own test set before you change it. The paper's 60 is a sensible default for a first build.

Every technique except cleanup adds at least one model call, and some add several searches. Wait matters most in a chat, because the customer waits for the rewriter on every message.

The table shows rough numbers for the made-up retailer. Wait is in milliseconds (ms), thousandths of a second. Cost is in paise, hundredths of a rupee.

Money follows the same shape, because model services usually charge by input and output tokens. If all 3,000 daily questions needed a history rewrite at 5 paise, that would be Rs 150 a day; only follow-ups do, so the real bill is lower. HyDE writes many more output tokens than a short query, so it costs more per question.

You do not have to pay for every technique on every message. The diagram shows a routing rule that sends each message down the cheapest path that fits, starting at the top.

Run the speaker chat through it. It is a follow-up, so it gets a history rewrite. The rewrite names the product and asks one thing, so it goes straight to search.

You run the same real questions through search with and without rewriting, then compare recall at 5. In this guide, recall at 5 is the share of test questions with at least one right chunk in the top 5. Some teams call this hit rate. Standard recall at k counts the share of all right chunks found; with one right chunk per question, as in the toy test below, the two are the same.

A test set is a fixed list of real questions, each with the chunks that should be found, run again after every change. The function below computes recall at 5. It needs no installs and was tested on Python 3.11, 3.12 and 3.13.

def recall_at_k(results, right_chunks, k=5):
    found = sum(1 for q in results if right_chunks[q] & set(results[q][:k]))
    return found / len(results)

right_chunks = {"q1": {"water-warranty"}, "q2": {"opened-items"},
                "q3": {"emi-options"}, "q4": {"wall-mounting"}}
before = {"q1": ["charging-port", "battery-tips"],
          "q2": ["earbud-care", "opened-items"],
          "q3": ["tv-sizes", "delivery-times"],
          "q4": ["wall-mounting", "tv-sizes"]}
after = dict(before, q1=["water-warranty", "charging-port"])

print(f"before: {recall_at_k(before, right_chunks):.0%}")
print(f"after:  {recall_at_k(after, right_chunks):.0%}")

# Output:
# before: 50%
# after:  75%

What just happened? Before, two of the four questions had a right chunk in the top 5, so recall is 50%. After a rewrite fixed the speaker question, three of four did, so it is 75%. A real test set should hold hundreds of questions, not four.

Here is how the made-up retailer built its real test set. Each step is cheap, and the order matters.

1. 1

   Collect real questions

   Pick about 300 real customer questions, including the 140 misses, follow-ups and Hinglish messages. Invented questions tend to be too clean.

2. 2

   Mark the right chunks

   For each question, the support team writes down which articles or chunks should be found.

3. 3

   Measure the starting point

   Run every question through search with no rewriting and record recall at 5.

4. 4

   Add one technique at a time

   Turn on history rewriting, run the set again and compare. Then try multi-query, then HyDE, so you know what helped.

5. 5

   Check answers, wait and cost

   Read a sample of final replies, since better chunks should mean better answers. Record extra wait and cost next to recall.

6. 6

   Try it on real traffic

   Send a small share of real chats through the new path. Watch how often those customers still ask for a person.

Here are the retailer's results (made up). Recall at 5 was 62% with no rewriting. History rewriting raised it to 74%, and adding multi-query raised it to 81%. HyDE on top added 1 more point for about 1.5 seconds of wait, so the team kept it off.

In this made-up run the techniques overlapped. Once the question was clear and searched in several wordings, HyDE had little left to fix.

Explain it as a fix for a measured problem. Show where search fails, add the cheapest rewrite that fixes it, and prove it with recall on a test set. Interviewers want trade-offs, not a list of every technique.

1. 1

   Name the failure

   Give one example of a bad search query, like a follow-up that only makes sense with the chat.

2. 2

   Place the rewriter

   Draw it between the user message and retrieval. The answer step still sees the original question.

3. 3

   Start cheap

   Cleanup in plain code first, then history rewriting with a small model.

4. 4

   Match each technique to a problem

   Multi-query with RRF for word mismatch, decomposition for compound questions, step-back for very specific ones, HyDE for questions shaped nothing like the documents.

5. 5

   Budget time and money

   Give a wait budget for the rewriter, and send only messy messages down the expensive paths.

6. 6

   Prove it

   Show recall at 5 before and after on a test set, and a log of every rewrite next to the original.