RAG Reranking: Cross-Encoders Explained Simply

The bot answers from the wrong text. The fast first search did find the right chunk, but placed it at rank 14, and the bot only passes the top 5 to the model. RAG reranking is the usual fix: a second, careful pass that reorders the results before the model reads them.

Picture a common setup. A train-ticket booking app runs a help bot for passengers' questions about ticket rules and refunds. Behind it sits the app's own rule book of about 1,200 pages. The setup is made up, but the problem is real.

One evening a passenger asks the bot a simple question. "My train was 4 hours late, so I did not travel. Can I get a full refund?" The bot replies that a cancellation charge of Rs 120 (rupees) will be cut. That is wrong.

The rule book's section 7.3 has the answer. If a train runs more than 3 hours late and you do not travel, you get the full fare back. The team sees about 40 complaints a week like this one.

The search log shows what happened. The search did find section 7.3, but at rank 14. The top 5 went to chunks about normal cancellation charges, which share words like refund, cancel and travel with the question.

You need six words: chunk, embedding, vector search, rank, top-k and reranker. RAG, short for retrieval-augmented generation, means the bot first searches your documents and then gives the best matches to the model. The model is the AI program that reads text and writes the reply.

Chunk. A chunk is a small piece of a long document, often a few paragraphs. The team splits its rule book into about 6,000 chunks. Section 7.3 on late trains is one chunk.

Embedding. An embedding is a list of numbers that captures the meaning of a piece of text. Texts with similar meanings get similar numbers. For example, "train delayed" and "train running late" get embeddings that sit close together.

Vector search. A vector is another name for that list of numbers. Vector search finds the chunks whose embeddings sit closest to the question's embedding. The bot asks a vector database, a store built to find nearby vectors fast, for the nearest chunks.

Rank and top-k. The rank is a chunk's position in the results, and rank 1 is the best match. Top-k means the first k results, so top 5 means ranks 1 to 5. Section 7.3 had rank 14, outside the top 5.

Reranker. A reranker is a second model that rereads a short list of search results and puts them in a better order. You will also see the names re-ranker, second-stage ranker and two-stage retrieval for the same plan.

RAG reranking is a careful second ranking step between the search and the model. The first stage finds many possible chunks fast, and the second stage reorders them so the best few reach the model. This plan is called two-stage retrieval.

Think of campus placements at an engineering college where 6,000 students apply. In round one, a recruiter shortlists 50 from their CVs in minutes. In round two, an interviewer talks to each of those 50 about this exact job. Nobody can interview 6,000 students, and nobody should hire from CVs alone.

Map it part by part. The CV shortlist is stage 1, the fast search. The interview is stage 2, the reranker. The job is the passenger's question.

Where the picture breaks: an interviewer remembers earlier candidates and compares them. A cross-encoder scores each question and chunk pair on its own, with no memory of the other 49.

The diagram below shows the help bot with a reranker added. Follow the question from left to right. Times are in ms, short for milliseconds (thousandths of a second).

Notice that the vector search itself does not change. The team only asks it for 50 results instead of 5, and adds one step after it. That makes reranking a cheap upgrade to try.

A 2021 benchmark study tested 10 search methods on 18 public datasets, scoring them on data they were not trained on. Reranking was among the best on average, but at a high computing cost.

A bi-encoder reads the question and each chunk separately, while a cross-encoder reads them together. That one difference makes the bi-encoder fast enough to search everything and the cross-encoder sharp enough to order a short list.

How a bi-encoder works

An encoder is a model that turns text into numbers. A bi-encoder runs the question and each chunk through encoders separately, and the two sides never see each other. The help bot's first-stage search is a bi-encoder.

The big win is timing. Chunks do not change between questions, so the team turns all 6,000 chunks into vectors once and stores them. When a question arrives, only the question needs new work.

A 2020 paper on dense passage retrieval, meaning search with embeddings, shows the scale this allows. It used two separate encoders and stored vectors for about 21 million passages. Making those vectors took about 8.8 hours on 8 GPUs, chips built to do many small sums at once.

That work happens before any question arrives. At question time, the paper's in-memory vector index answered 995 questions per second, returning the top 100 passages for each.

The cost is detail. Each chunk is squeezed into one list of numbers before anyone knows the question. The vector for section 7.3 cannot know that this passenger cares about "4 hours late" and "did not travel". So close cousins, like general cancellation chunks, can score just as high.

How a cross-encoder works

A cross-encoder joins the question and one chunk into a single input and returns one number: how well this chunk answers this question. Because it reads both at once, every word of the question can be weighed against every word of the chunk.

So the cross-encoder can notice that section 7.3 is about a late train and a passenger who did not travel. The cancellation chunks are about a passenger who chose to cancel. That is the round-two interview from the placement story.

The price is speed. Nothing can be prepared ahead of time, because the score depends on the question. To rank all 6,000 chunks, the cross-encoder would need 6,000 full model runs for every question.

A 2019 paper on scoring sentence pairs measured this. Its cross-encoder took about 21.7 seconds on an 80-core server processor to score 1,000 candidates for one input. A bi-encoder with stored candidate vectors took 115 ms on the same processor.

On one GPU, the same test took about 2.6 seconds against 19 ms. The paper found cross-encoders often score better but called them too slow for practical use. Each extra candidate needs its own full pass, so they only get a short list.

The diagram below puts the two side by side. Follow the rule chunk down each path.

The table below compares the three kinds of ranking you will meet in this guide. The third, a large language model used as a reranker, gets its own section later.

You ask the first search for more results, score every question and chunk pair in one batch, sort by score and keep the best few. The function below does exactly that, and the whole block runs as shown.

The score_pairs argument is your model's batch scoring call. It takes a list of question and chunk pairs and returns one score per pair. Here a stand-in returns fixed scores, so you can run the code without a model.

def rerank(question, candidates, score_pairs, keep=5):
    pairs = [(question, chunk["text"]) for chunk in candidates]
    scores = score_pairs(pairs)            # one batched call, one score per pair
    ranked = sorted(zip(scores, candidates),
                    key=lambda item: item[0], reverse=True)
    return [chunk for score, chunk in ranked[:keep]]


candidates = [                             # stage 1 output, in search order
    {"id": "4.1", "text": "Cancellation charges for confirmed tickets"},
    {"id": "4.2", "text": "How to cancel a ticket and get a refund"},
    {"id": "7.3", "text": "Full refund if the train is over 3 hours late and you do not travel"},
]


def stand_in_scores(pairs):                # swap in your cross-encoder's batch call
    return [0.12, 0.31, 0.97]


question = "My train was 4 hours late and I did not travel. Full refund?"
best = rerank(question, candidates, stand_in_scores, keep=2)
print([chunk["id"] for chunk in best])

Run it and you should see ['7.3', '4.2']. Section 7.3 was last in the search order and is now first. The key tells Python to compare scores only, and reverse=True puts the highest score first.

In the real bot, stage 1 returns 50 candidates and keep is 5. The rest of the bot does not change.

Retrieve enough that the right chunk is almost always in the list, then keep only as many as the model needs to answer. A common start is about 50 retrieved and 3 to 8 kept, but your own test questions should decide.

The number you retrieve sets the ceiling. The team tried 20, 50 and 100 on 200 test questions (numbers made up). The right chunk was in the top 20 for 170 of them, in the top 50 for 190, and in the top 100 for 194.

That share, 190 of 200 or 0.95, is called recall at 50. Going from 50 to 100 doubled the reranking work for 4 more questions. The team chose 50.

The number you keep is a different choice. The team's first idea was a shortcut: skip reranking and send the top 20 chunks to the model. The right rule got included more often, but answers became muddled and each question cost more.

Research explains part of this. A 2023 study, Lost in the Middle, gave models 10, 20 or 30 documents with the answer in one of them. Accuracy was usually highest when the answer sat at the start or end, and lower in the middle.

For one closed model, the worst middle position in 20 or 30 documents scored below giving no documents at all. In the paper's search test, going past 20 retrieved documents improved answers only slightly.

Here is a simple way to pick both numbers:

1. 1

   Build a small test set

   Collect 100 to 200 real passenger questions. Someone who knows the rules, such as a support lead, marks the chunk that truly answers each one.

2. 2

   Measure the ceiling

   For fetch sizes of 20, 50 and 100, count how often the right chunk appears anywhere in the list.

3. 3

   Pick the smallest fetch that is high enough

   Choose the smallest size where adding more barely helps. For the help bot that was 50.

4. 4

   Try a few keep sizes

   Run 3, 5 and 8 kept chunks through the full bot. Check which gives correct answers most often.

5. 5

   Check the wait time

   Make sure the bigger fetch still fits your time budget, which the next section works out.

Reranking adds the time to score every retrieved chunk, from tens of milliseconds to seconds depending on hardware, plus the cost of running the reranker. It can also save money, because the model then reads fewer chunks.

Latency means how long one request takes, from question to reply. For a real anchor, the 2021 benchmark study estimated about 450 ms per question for a small six-layer cross-encoder reranking 100 keyword-search hits on one GPU. A layer is one stacked processing step inside a model; more layers usually mean a bigger, slower model.

On an 8-core server processor, the same setup took about 6.1 seconds. The paper does not say whether these figures include the keyword search itself. Your numbers depend on hardware, chunk length and model size, so measure your own.

The time maths

These are the help bot's numbers, all made up but realistic:

• Stage 1 vector search for the top 50: about 30 ms.
• Cross-encoder scoring 50 pairs in one batch on one GPU: about 200 ms.
• Model writing the answer from 20 chunks: about 1.8 seconds.
• Model writing the answer from 5 chunks: about 1.4 seconds, because it has less to read.

The 20-chunk shortcut takes 30 ms plus 1.8 seconds, about 1.83 seconds. The reranked setup takes 30 ms plus 200 ms plus 1.4 seconds, about 1.63 seconds. Here the full reply got faster even though a step was added.

The money maths

Most model services charge by tokens. A token is a small piece of text, roughly a word or part of a word. Say each chunk is about 300 tokens, and the price is Rs 0.08 for every 1,000 tokens the model reads (made up).

Twenty chunks are 6,000 tokens, so Rs 0.48 per question. Five chunks are 1,500 tokens, so Rs 0.12. At 20,000 questions a day, that is Rs 9,600 a day for 20 chunks against Rs 2,400 for 5.

Add Rs 3,000 a day for the reranker server (made up), and the reranked total is Rs 5,400. That is Rs 4,200 a day less than the 20-chunk shortcut. Against the original 5-chunk bot, reranking costs Rs 3,000 a day more, in exchange for better answers.

Yes, you can ask a large language model (LLM) to judge which chunks best answer the question, and use its judgement as the new order. It can be more flexible than a small cross-encoder, but it is slower and costlier, so it usually fits a short list or only the hard questions.

An LLM is the same kind of model that writes the bot's answers. There are three common ways to ask it to rank:

• Pointwise: show the model one chunk at a time and ask for a score from 0 to 10. Simple, but a 7 in one request may not mean the same as a 7 in another.
• Listwise: show the model 20 numbered chunks at once and ask for the best order as a list of numbers. One request, but a long prompt.
• Pairwise: show two chunks and ask which one answers the question better. Repeat for many pairs and combine the results.

A 2023 paper argued that ready-made large models find the pointwise and listwise styles hard. Pointwise needs scores that mean the same thing across separate requests. Listwise answers sometimes left chunks out or repeated them.

Its pairwise method reranked the top 100 passages from a keyword search. With an open model of 20 billion parameters (the numbers a model learns in training), it compared well with the previous best method on two passage-ranking test sets. That method used a closed model estimated to be 50 times larger.

Comparing every pair needs a number of requests that grows with the square of the list. The authors also proposed faster versions based on sorting and on a sliding window.

For the help bot, a listwise request on the top 20 would add about 2 seconds and Rs 0.30 per question (made up). Across 20,000 questions a day, that is Rs 6,000 more. The team keeps the cross-encoder for all questions and tests the large model only on hard refund questions.

The chart below is how the team decides which reranker, if any, to use. Start at the top and follow the answer that fits your bot.

You run the same test questions with and without the reranker and compare where the right chunk lands. Two simple numbers do most of the work: hit rate at 5 and mean reciprocal rank.

Hit rate at 5. This is the share of test questions where the right chunk appears in the top 5. If it appears for 88 of 100 questions, the hit rate at 5 is 0.88.

Mean reciprocal rank, or MRR. Reciprocal rank is 1 divided by the rank of the right chunk. Rank 1 gives 1, rank 2 gives 0.5, rank 4 gives 0.25, and missing gives 0. MRR is the average over all test questions.

The code below works out both numbers. Each test question gives the ranked chunk ids your search returned and the id marked as correct. The block runs as shown, on three questions before and after reranking.

def hit_rate_and_mrr(results, k=5):
    """results: one (ranked_ids, correct_id) pair per test question."""
    hits, reciprocal_total = 0, 0.0
    for ranked_ids, correct_id in results:
        top = ranked_ids[:k]
        if correct_id in top:
            hits += 1
            reciprocal_total += 1 / (top.index(correct_id) + 1)
    count = len(results)
    return round(hits / count, 2), round(reciprocal_total / count, 2)


before = [(["4.1", "4.2", "2.8", "4.5", "1.9"], "7.3"),   # missing from top 5
          (["5.2", "6.1", "5.4", "3.3", "6.0"], "6.1"),   # rank 2
          (["3.1", "3.4", "2.2", "8.0", "8.1"], "3.1")]   # rank 1
after = [(["7.3", "4.2", "4.1", "2.8", "4.5"], "7.3"),    # rank 1
         (["6.1", "5.2", "6.0", "5.4", "3.3"], "6.1"),    # rank 1
         (["3.4", "3.1", "2.2", "8.0", "8.1"], "3.1")]    # rank 2
print(hit_rate_and_mrr(before), hit_rate_and_mrr(after))

Run it and you should see (0.67, 0.5) (1.0, 0.83). Before reranking, one question missed the top 5, so the hit rate is 2 of 3. After, every right chunk is in the top 2.

Python's index starts at 0, so the + 1 turns a position into a normal rank. Because only the top 5 count, a chunk at rank 14 adds 0, just like a missing one. Engineers call this version MRR at 5.

On the team's 200 test questions, the old search scored a hit rate of 0.61 and MRR of 0.42 (made up). With reranking, it scored 0.88 and 0.74.

Ranking numbers are not the whole story. Keep an eye on these too:

• Answer quality: read 50 sample replies before and after, and mark each one right or wrong.
• Slow requests: track p95 latency, the time within which 95 of every 100 questions get a reply.
• Daily cost: model tokens plus the reranker server, compared with last month.
• Fallbacks: how often the reranker failed or ran late and the bot used the first-stage order instead.

After the change, the late-train complaints drop from about 40 a week to about 6. The team adds every new complaint question to the test set, so the next change is checked against mistakes passengers already saw.

Three things catch teams most often: long chunks that are silently cut, a reranker that does worse on your kind of text, and a slow step with no fallback. Each one has a cheap check.

Long chunks are cut without an error

A cross-encoder has a maximum input length, counted in tokens, for the question and chunk together. One widely used open-source reranking library cuts longer pairs to fit by default, with no error. That is true of its release current on 19 October 2026, read in its source code, so pin your version and recheck after upgrades.

It trims the longer of the two texts first, which is usually the chunk. So a rule at the end of a long chunk may never be read. Check it: compare your longest chunks, in tokens, with the model's documented maximum input length.

A reranker can do worse on unfamiliar text

A reranker trained on very different text can push a good chunk down. Railway rule codes look nothing like web search questions. Only your own test set can tell you whether the reranker helps your documents.

Plan for a slow or broken reranker

Give the reranking call a time limit, such as 300 ms. If it fails or runs late, fall back to the first-stage order. A slightly worse answer beats a passenger staring at a spinner.

The code below adds that limit. It reuses rerank, candidates and question from the earlier block, so run the two blocks together.

import time
from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(max_workers=4)


def rerank_or_fallback(question, candidates, score_pairs, keep=5, limit_s=0.3):
    future = pool.submit(rerank, question, candidates, score_pairs, keep)
    try:
        return future.result(timeout=limit_s)
    except Exception:                      # timed out, or the reranker failed
        future.cancel()                    # drop it if it has not started yet
        return candidates[:keep]           # first-stage order


def slow_scores(pairs):                    # a reranker that takes 1 second
    time.sleep(1)
    return [0.12, 0.31, 0.97]


start = time.perf_counter()
best = rerank_or_fallback(question, candidates, slow_scores, keep=2)
print([chunk["id"] for chunk in best], round(time.perf_counter() - start, 1), "s")

Run both blocks and the last line should read ['4.1', '4.2'] 0.3 s: the first-stage order, after the 300 ms limit. A call that has already started cannot be cancelled and keeps running. The future.cancel() line drops calls still waiting in the queue, so a burst of slow requests does not pile up work. Count fallbacks so you notice a slow reranker.

Explain it as a two-stage search: a cheap stage that finds candidates and an expensive stage that orders them. Then show you can pick the numbers, pay for it and measure it.

1. 1

   Name the problem

   Say that vector search often finds the right chunk but ranks it too low, like section 7.3 at rank 14.

2. 2

   Draw two stages

   Draw a bi-encoder search returning 50 candidates, then a cross-encoder reranker keeping the best 5.

3. 3

   Explain why two stages

   Chunk vectors can be made ahead of time, so stage 1 scales. A cross-encoder cannot, so it only sees a short list.

4. 4

   Pick the numbers

   Say how you would choose how many to fetch and keep, using a test set and recall at 50.

5. 5

   Do the cost and time maths

   Add up the milliseconds and rupees out loud, including the tokens the model no longer reads.

6. 6

   Add a fallback and a metric

   Mention a time limit with fallback to the first-stage order, and hit rate at 5 plus MRR before and after.