Semantic Routing: Sending Queries to the Right Source

It answers from the wrong source. A timing question, a membership date and a refund rule live in three different places, and a bot that searches only one of them guesses for the other two.

Picture a common setup. A gym chain with 14 branches in Hyderabad runs a support bot inside its members' app, and it handles about 4,200 chats a day. The setup is made up, but the problem is real.

The bot can use three kinds of knowledge. An FAQ list holds about 180 short, fixed answers, like opening hours and parking. A SQL database (tables you ask for rows with SQL, a query language) holds each member's plan, end date and personal training sessions.

A third source is a set of policy documents about freezes, refunds and branch transfers. The first version of the bot put the FAQ and the policy documents into one search index (a store built for fast text search) and searched it for every question.

That design is called RAG, short for retrieval-augmented generation: the bot fetches related text, then a language model (an AI model that generates text) writes the answer from it. The member database was not in the index at all.

The support team reads 300 saved chats from one week and finds three kinds of trouble:

• A member asks when her membership ends. The bot quotes the general renewal policy, because no document holds her personal end date.
• A member asks for the Madhapur branch's Sunday hours. The bot quotes a long paragraph about holiday closures instead of the one-line FAQ answer.
• A member asks whether he can freeze his plan for a knee surgery. The bot gives the short FAQ line and misses the medical freeze rules in the policy document.

Out of 300 chats, 41 had gone to the wrong place. The team asks for one fix: send each question to where its answer actually lives.

That fix is a router, a step that sorts questions before anything answers them. The rest of this guide builds one for the gym chain's bot, keeps its database route safe and tests it.

Semantic routing is a step that compares the meaning of a question with example questions for each route, and sends it down the closest one. A route is one path to one source, such as the FAQ.

The word semantic means it works on meaning, not exact words. "When does my plan expire?" and "Till what date is my membership valid?" share almost no words, yet both belong on the account route.

Think of the help desk at a municipal corporation office. You explain your problem in your own words. The person at the desk does not solve it; she listens, then sends you to the right window.

Where the picture breaks: the desk person understands you. The router understands nothing; it only measures how close two lists of numbers are, so it is only as good as its examples.

Two terms do all the work. An embedding is a list of numbers, made by an embedding model, that stands for the meaning of a text. Texts with similar meanings get lists that point in similar directions.

Cosine similarity is a score for how closely two embeddings point the same way. It is 1 when they point exactly the same way and lower as meanings drift apart. The exact scores depend on the embedding model.

A 2019 paper by Reimers and Gurevych made this practical for sentences. Its authors report that finding the most similar pair among 10,000 sentences fell from about 65 hours to about 5 seconds.

The trick was to embed each sentence once and then compare the numbers with cosine similarity, instead of running a large model on every pair. A router uses the same trick.

Here are the gym chain's routes. Each box on the right is one place a question can go. Notice that one route leads to people, not to data.

It works in two phases. Once, ahead of time, you embed example questions for each route. Then, for every new question, you embed it and pick the route whose examples sit closest.

Some tools call these examples utterances. The team takes about 30 per route from old chats, written the way members really type.

1. 1

   Collect example questions

   Pull about 30 real questions per route from past chats. Keep the short forms, the typos and the mixed Hindi and English.

2. 2

   Embed them once

   Each example becomes an embedding, stored with its route name. This cost is paid once at start-up, not on every chat.

3. 3

   Embed the new question

   When a member writes, the router makes one embedding for the message. That is the only model call the router needs.

4. 4

   Score every route

   Compare the question with every example and keep the best score for each route.

5. 5

   Apply the threshold and the margin

   If the best score clears the threshold and beats the runner-up clearly, the question goes down that route. Otherwise it goes down the unsure path.

The code below is a complete router you can run with Python and numpy (a library for arrays of numbers). Its embed function is a stand-in that counts three-letter pieces of text, so it needs no model download. Beginners can skim the code; the output and the notes after it carry the idea.

Pause and predict before you run it. The last question, "mera plan kab khatam hoga", is Hindi typed in English letters for "when will my plan end". Which route will the stand-in pick?

import zlib
import numpy as np

def embed(text, dims=512):
    # Stand-in for an embedding model: it counts three-letter pieces,
    # so it matches spelling, not meaning. Swap in a real model.
    v = np.zeros(dims)
    t = " " + text.lower() + " "
    for i in range(len(t) - 2):
        v[zlib.crc32(t[i:i + 3].encode()) % dims] += 1
    return v / np.linalg.norm(v)

ROUTES = {
    "faq": ["What time does the gym open?", "Is there parking at the gym?"],
    "account": ["When does my membership end?", "How many PT sessions are left?"],
    "policy": ["Can I freeze my plan for a surgery?", "What is the refund rule?"],
}
ROUTE_VECTORS = {name: np.array([embed(t) for t in examples])
                 for name, examples in ROUTES.items()}
THRESHOLD = 0.45  # set for this stand-in only; tune it for your model
MARGIN = 0.05

def pick_route(question, threshold=THRESHOLD):
    q = embed(question)
    scores = {name: float(np.max(vecs @ q))
              for name, vecs in ROUTE_VECTORS.items()}
    first, second = sorted(scores.values(), reverse=True)[:2]
    best = max(scores, key=scores.get)
    if first < threshold or first - second < MARGIN:
        return "unsure", round(first, 2)
    return best, round(first, 2)

for question in ["when does my membership end",
                 "what time does the gym open on sunday",
                 "can i freeze my membership",
                 "mera plan kab khatam hoga"]:
    print(question, "->", pick_route(question))

Tested on Python 3.13.16 with numpy 2.5.3, it prints exactly this:

when does my membership end -> ('account', 0.95)
what time does the gym open on sunday -> ('faq', 0.83)
can i freeze my membership -> ('policy', 0.56)
mera plan kab khatam hoga -> ('unsure', 0.19)

What just happened? The first three questions share spelling with an example, so they clear the 0.45 threshold. The Hindi question shares no spelling with any example, so it scores 0.19 and goes down the unsure path.

That is the stand-in's limit, not the method's. An embedding model that has learned Hindi typed in English letters can place it near the account examples; add such questions to your test set to check yours. Replace embed with a call to your model and keep everything else.

The line with @ scores the question against every example of a route at once, and np.max keeps the closest one. That is one choice among several, and the next paragraph shows another.

One open-source routing library, in version 1.1.0, has an indexed router that works differently. It takes the 5 closest examples overall and by default averages each route's scores among them; max and sum are options. We read this in its source code, as of 19 October 2026, so pin your version.

An average rewards a route with several close examples, while the closest example rewards one sharp match. Neither is right everywhere, so run both on your labelled questions.

Notice the comment on THRESHOLD. The value 0.45 suits this stand-in and nothing else. A real model gives different scores, so its threshold comes from its own test, as the next section shows.

Now follow one chat through the production router, which uses a real embedding model. Watch the box with the question mark: it alone decides between sending the chat on and asking the member.

You choose it by running the router on a few hundred labelled questions at several thresholds. Pick the value where wrong routes become rare without sending too many chats down the unsure path.

No single number works everywhere. Scores depend on the embedding model, the language and how close your routes are. That is why the stand-in above uses 0.45 and the gym chain's real model uses 0.62.

The team labels 400 recent questions by hand with the right route. Then it runs the router at three thresholds and counts two things: wrong routes and unsure chats. These counts are made up.

Then the team weighs the two mistakes. A wrong route gives a confident wrong answer, such as a wrong membership date. An unsure chat costs the member one extra tap, or a few minutes of a support agent's time.

A wrong date is much worse for the business, so they accept more unsure chats and settle on 0.62. Going to 0.70 would halve wrong routes but more than double the unsure chats.

Check the gap between the top two routes

A question can score high on two routes at once. "Can I transfer my membership to my brother?" scores close on both account and policy. A margin rule handles this: if the top two scores are within 0.05, treat the question as unsure.

The runnable router above already has this rule, in the line that compares first and second. Without it, a 0.70 against 0.69 split would be settled by noise.

It should ask the member one short clarifying question or hand the chat to a person. It should never guess, because a guess looks just as confident to the member as a right answer.

The gym chain's unsure path has four parts:

• Ask a choice question. The bot shows two buttons built from the top two routes, such as My plan dates and Freeze rules.
• Split a two-part message. A member who asks to renew a plan and for Sunday timings gets each part routed on its own, then one combined reply.
• Hand over. After one unclear reply, the chat goes to the support team with the top two routes attached, so the agent starts with context.
• Log it. Every unsure question is saved, because today's unsure questions become tomorrow's route examples.

The diagram shows the two checks in order. Notice that both checks fail into the same place: a choice question, not a guess.

Route it to a fixed set of named, read-only queries, with the member's id taken from the login. Never let the bot run SQL text written by a language model or typed by a member.

Text-to-SQL means a language model writes a SQL query from a question. It is tempting on a support bot, and risky. A member could type words that steer the model into reading another member's row or changing data.

That steering is called prompt injection. The OWASP guidance on it advises handling privileged functions in code rather than giving them to the model. OWASP is a non-profit that publishes security guidance.

So the account route stays boring on purpose. A second, smaller router picks one named query, called the intent, such as membership_end. The SQL text is fixed in code, and only the member's id goes in, as a parameter.

A parameter is a value you hand to the driver apart from the SQL text. The driver sends it separately or escapes it safely, so it is never read as SQL. The OWASP cheat sheet on SQL injection lists these parameterised queries as its first defence.

import psycopg

ACCOUNT_QUERIES = {
    "membership_end":
        "SELECT plan_name, end_date FROM memberships WHERE member_id = %s",
    "sessions_left":
        "SELECT sessions_left FROM pt_packs WHERE member_id = %s",
}

def run_account_query(intent, member_id, dsn):
    if intent not in ACCOUNT_QUERIES:
        return None  # no SQL runs; the bot asks the member to choose
    with psycopg.connect(dsn) as conn:
        return conn.execute(ACCOUNT_QUERIES[intent], (member_id,)).fetchone()

It uses psycopg 3, a PostgreSQL driver for Python. The intent check comes first: if the router returns any name outside ACCOUNT_QUERIES, no SQL runs and the bot falls back to the choice question.

The member_id passed in comes from the login session, never from the chat text. So a member who asks for a friend's plan still gets only their own row.

We ran it on PostgreSQL 16.15 with psycopg 3.3.6 against a small test table. The calls and what they returned:

membership_end, 1042 -> ('Gold 12 months', datetime.date(2027, 3, 31))
sessions_left, 1042 -> (7,)
delete_everything, 1042 -> None
membership_end, '1042 OR 1=1' -> InvalidTextRepresentation: invalid input syntax for type integer: "1042 OR 1=1"

The last line shows a parameter staying data. Even a crafted string reaches the database as a value, and PostgreSQL rejects it as a bad number instead of running it as SQL.

Scope this to the driver. Psycopg 3 sends the query and the parameters to the server separately; the server log showed the query with $1 and the value listed apart. Psycopg 2.9.13, which we also checked, merges the escaped values into the query text on the client instead.

Lock the database account, not just the session

Add a second guard inside the database. Give the bot its own database user that can only read two tables, and make every transaction it opens read-only by default.

CREATE ROLE support_bot LOGIN;
GRANT SELECT ON memberships, pt_packs TO support_bot;
ALTER ROLE support_bot SET default_transaction_read_only = on;

A transaction is a group of database commands that succeed or fail together. In PostgreSQL, a read-only transaction refuses INSERT, UPDATE and DELETE on non-temporary tables, and every CREATE, ALTER and DROP command. The setting default_transaction_read_only makes each new transaction read-only unless the session says otherwise.

To check it, connect as support_bot with psql, PostgreSQL's command-line client, and run these lines:

SHOW default_transaction_read_only;
SELECT plan_name, end_date FROM memberships WHERE member_id = 1042;
DELETE FROM memberships;
SET default_transaction_read_only = off;
DELETE FROM memberships;
SELECT * FROM payments;

On PostgreSQL 16.15, the first two lines print on and the member's row. The other four lines print this, in order:

ERROR:  cannot execute DELETE in a read-only transaction
SET
ERROR:  permission denied for table memberships
ERROR:  permission denied for table payments

Look at the line that says SET. The bot's own session switched read-only off with a plain SET, and nothing stopped it. BEGIN READ WRITE does the same for one transaction; we tested both.

So default_transaction_read_only is a default, not a lock. The real lock is the GRANT: among the gym's tables, the bot may only SELECT from these two. It can still read every row in them, so the named queries decide which member it sees. OWASP's cheat sheet gives the same advice: read-only accounts get read access only, to the tables they need.

Use keyword rules for a few exact phrases, semantic routing for most traffic, and a model classifier only for the messy leftovers. A model classifier means asking a language model to read the question and reply with a route name.

Each option handles new wording better than the one before it, and does more work per chat to get there. The table compares them on the gym chain's traffic.

The gym chain layers them. A short keyword list catches urgent phrases, such as cancel my membership, and sends them straight to the support team. Semantic routing takes everything else.

A model classifier is a fair last try for unsure chats, before the bot asks the member. It costs a model reply, but only on the small share of chats that need it.

Research routes a related choice. Adaptive-RAG, a 2024 paper, trains a smaller language model as a classifier that predicts how complex each question is. It then picks no retrieval, a single search or step-by-step retrieval.

The authors report better overall efficiency and accuracy than the baselines they compared, on open-domain question-answering datasets. That is routing by difficulty, not by source, but the shape is the same: a cheap decision before the expensive work.

A router decides once, up front, before any search runs. If a question needs several searches with checks in between, that is a loop, often called agentic RAG. A router in front can send only the hard questions into it.

Test it like any classifier: keep a labelled set of real questions, run the router on it after every change, and read the mistakes. A classifier is any program that sorts inputs into labels.

The most useful view is a confusion matrix, a grid that counts how often each true route was given each predicted route. In scikit-learn, a Python library for machine learning, rows are the true routes and columns the predicted ones.

Run this after the router code above. It repeats the threshold test on seven labelled questions, then prints the confusion matrix at the router's own threshold.

from sklearn.metrics import confusion_matrix

TEST_SET = [
    ("is there parking", "faq"),
    ("gym timings on sunday", "faq"),
    ("how many sessions do i have left", "account"),
    ("till what date is my membership valid", "account"),
    ("can i freeze my membership", "policy"),
    ("can i transfer my membership to my brother", "policy"),
    ("refund policy", "policy"),
]
truth = [label for _, label in TEST_SET]

for threshold in (0.3, 0.45, 0.6):
    predicted = [pick_route(q, threshold)[0] for q, _ in TEST_SET]
    wrong = sum(p not in (t, "unsure") for p, t in zip(predicted, truth))
    print(threshold, "wrong:", wrong, "unsure:", predicted.count("unsure"))

predicted = [pick_route(q)[0] for q, _ in TEST_SET]
print(confusion_matrix(truth, predicted,
                       labels=["faq", "account", "policy", "unsure"]))

With scikit-learn 1.9.1 it prints:

0.3 wrong: 1 unsure: 1
0.45 wrong: 0 unsure: 4
0.6 wrong: 0 unsure: 5
[[1 0 0 1]
 [0 1 0 1]
 [0 0 1 2]
 [0 0 0 0]]

What just happened? At 0.3, the transfer question went wrongly to account. At 0.45 nothing went wrong, but four of seven chats were unsure. That is the same trade as the table above, in miniature.

Read the matrix row by row. The fourth column is unsure: one FAQ, one account and two policy questions landed there. The fourth row is empty because no question is labelled unsure. The unsure account question is the membership one from the start of this guide; the stand-in matches spelling, so it missed it.

In the gym chain's real matrix, most mistakes sat between policy and FAQ. Short FAQ lines about freezes looked a lot like the freeze policy itself. The fix was in the content, not the code:

• Give each topic one home. The team cut FAQ entries that only summarised a policy, so freezes now live only in the policy route.
• Add examples from mistakes. Every wrong route in the weekly sample becomes a new example for the right route.
• Remove confusing examples. An example that sits close to two routes does more harm than good.
• Watch the unsure rate. A sudden jump usually means members are asking about something new, like a new branch or offer.
• Retest after any model change. Scores shift with the embedding model, so the threshold must be tested again.

After four weeks, the weekly sample of 300 chats shows 9 wrong routes instead of 41, and about 6 percent of chats take the unsure path. Both numbers are made up, like the rest of the setup.

Most failures come from the routes and examples, not the maths. Overlapping sources, thin examples and new topics cause far more wrong routes than the similarity score does.

• Overlapping routes. If two sources hold the same topic, no threshold can fix it. Give each topic one home.
• Too few examples. Five neat examples per route miss slang and mixed languages. Use many real, varied ones.
• Long messages. A member pastes three paragraphs about a billing dispute, and the meaning blurs into one embedding. Route each clear question separately, or hand over.
• Drift. New plans, branches and offers bring questions no example covers. The unsure rate shows this first.
• Trusting the router with safety. A crafted message can push the router toward a route. Keep permissions inside each route and the database account read-only.

Log every routing decision: the message, the top three scores and the chosen route. When the support team flags a bad chat, you can see in seconds whether the router or the source was at fault.

Explain it as a cheap sorting step in front of several sources, and back each design choice with the failure it prevents. These six steps make a good frame for the answer.

1. 1

   Name the sources

   List where answers live: an FAQ for fixed answers, a database for personal data, documents for rules. Say why one search index cannot serve all three.

2. 2

   Draw the router

   Show the router in front of the sources. Say it embeds the question and compares it with example questions for each route.

3. 3

   Set the threshold from data

   Explain the trade between wrong routes and unsure chats, and how you would pick the number from labelled questions.

4. 4

   Design the unsure path

   Describe the margin rule, the clarifying question, the split for two-part messages and the handover to people.

5. 5

   Lock down the database route

   Named, fixed queries with parameters, the user's id from the session, and a database user granted SELECT on only the tables it needs.

6. 6

   Explain how you measure it

   A labelled test set, a confusion matrix, the unsure rate and a weekly review of real mistakes.