LLM Model Selection Per Task: Tiering, Routing and Cascades

You pay large-model prices for easy work, and users wait for answers a small model could give in a fraction of the time. Learning how to choose an LLM per task fixes both problems. It starts with a test, not a favourite model.

Picture a common setup. A grocery delivery app in Hyderabad uses AI for three jobs, all sent to the biggest model its provider offers. The setup is made up, but the problem is real.

The first job is order-issue sorting. When a customer writes "the curd packet is missing", the app tags the message so it reaches the right team. The five tags are missing item, late delivery, damaged, wrong item and other.

The second job is recipe suggestions from the items in your cart. The third is refund disputes: a customer says a refund decision was unfair, and the model reads the chat, the order and the policy, then suggests a decision.

It all worked. Then the monthly AI bill reached about Rs 3 lakh (300,000), and users complained that the recipe screen took 4 or 5 seconds to load.

Before changing anything, the team sorted 30 real customer messages by hand. 25 were easy, like "Milk packet missing from order 7712". Only 5 needed real thought, like one long message mixing a late delivery, a broken egg tray and a refund demand.

Most of the work was easy, but every request went to the most expensive model. The rest of this guide follows this team as it picks a model per task and decides where each request goes.

You need seven words. Each gets a plain meaning here, with an example from the grocery app.

• Model, or LLM (large language model): the AI program that reads text and writes text back. The instructions you send with the text are the prompt.
• Token: the unit a model reads and writes, a short piece of text such as a short word or part of a longer one. Hosted services usually bill per token (as of October 2026).
• Tier: a size level you choose to use, such as small, medium or large. The app puts issue sorting on the small tier.
• Pass mark: the score a model must reach on your test before you trust it with a task, such as 92 correct tags in 100.
• Router: the code that picks a tier for each request before any model answers. Example: every request from the recipe screen goes to the medium tier.
• Cascade: try the small model first, check its answer, and escalate (send the request up to a bigger model) only if the check fails.
• Fallback: the backup plan when a model call fails, for example because the service is down or too slow. Example: the message goes to a person's queue.

One picture helps with the last three words. Think of a clinic, mapped part by part in the table below.

Where the picture breaks: a physician knows when a case is beyond them. A small model often does not, and can give a wrong answer with full confidence. That gap comes back in the engineer sections.

A large model has more parameters than a small one. Parameters are the numbers a model learns during training. More of them usually makes a model better on hard, messy work, but slower and more expensive per request. On narrow, clear tasks, a small model often scores close to a large one.

In a standard dense model, unlike the mixture-of-experts kind below, almost all parameters do work for every token. A 2020 study of language model scaling puts that work at about two operations per parameter per token. A smaller extra term grows with context length, the amount of text the model reads at once.

That is why more parameters means more computing for every token, and usually a higher price and a longer wait. One exception matters: mixture-of-experts models use only part of their parameters for each input. For them, size on paper says less about cost.

Here is the rough pattern you are likely to see on your own tests. It is a tendency, not a rule:

• Small models do well on narrow jobs with a fixed set of answers: tagging a message, pulling an order number out of text, or a yes or no.
• Medium models suit short writing with a few rules, like a recipe that must use what is in the cart.
• Large models do better on long, messy inputs that need several steps of reasoning, like weighing a customer's story against a refund policy.

Every size can make confident mistakes, and a newer, smaller model can beat an older, larger one. A 2022 paper on compute-optimal training, getting the best model from a fixed training budget, shows this. Its 70-billion-parameter model used the same training compute (total computing spent on training) as a 280-billion one, but four times more data. It beat that model, and others of up to 530 billion parameters, on a large range of tasks.

So never pick by size alone. Pick by how a model scores on your task.

You write a pass mark for each task, test a few model sizes on real examples, and pick the cheapest one that passes. Every option is judged on quality, cost and speed.

Quality is how often the answer is one you would accept. Cost is what you pay per request, which usually grows with model size and with the amount of text. Speed is how long the user waits.

The team wrote one line per task before anyone tested a model. Each line gives the bar, the cost of a mistake and the daily volume:

• Order-issue sorting: right at least 92 times in 100. A wrong tag only sends the message to the wrong team, who pass it on. Runs about 3,000 times a day.
• Recipe suggestions: at least 34 of 40 recipes pass a checklist (uses cart items, respects the vegetarian setting, at most 8 steps). Runs about 8,000 times a day while users wait.
• Refund disputes: the suggested decision matches a senior support agent at least 54 times in 60. Mistakes cost money and trust. Runs about 400 times a day.

A task that is cheap to get wrong and runs often should lean small. A costly, rare task can afford a large model.

A test set is a list of real inputs, each with its right answer written down, that you run through every model you compare. Think of it as a mock exam with an answer key: every model sits the same paper.

The team built one with 200 cases across the three tasks. It took five steps:

1. 1

   Collect real inputs

   Pull 100 old customer messages, 40 real carts and 60 past refund disputes from the app's logs (saved records of past requests). Remove names and phone numbers first.

2. 2

   Include hard cases on purpose

   Make about one case in five tricky: messages mixing Telugu and English, nearly empty carts, and disputes with missing photos.

3. 3

   Write the right answers

   Support staff tag each message and a senior agent decides each dispute. Recipes have no single right answer, so the checklist is the answer key.

4. 4

   Run every model on the same cases

   Same prompt, same cases, three sizes. Record whether each answer passed, how long it took and what it cost.

5. 5

   Save the results

   Keep the scores in a file with the date and the exact model version, so later re-checks compare like with like.

Here are the results. Each cell shows the score, the cost per 1,000 requests and the average wait. To keep it simple, a request costs the same on every task. A cell marked picked is the cheapest tier that clears that task's bar.

Read it one row at a time. On sorting, the large model is 3 points better than the small one, but costs 45 times more and makes users wait 8 times longer. The small model clears the bar on this test, so it wins. A later engineer section shows how sure a score this close to the bar really is.

On refund disputes, only the large model clears the bar, so its price is worth paying. Researchers also judge models on more than one measure. A large 2022 evaluation of language models reported seven measures, including accuracy, robustness, fairness and efficiency, rather than one score.

Model tiering means giving each task one fixed tier, the cheapest that passed its test. It turns test results into a short written plan the whole team can follow.

Keep the number of tiers small. Two or three tiers are easy to explain and re-test. Ten tiers means ten things to re-check whenever something changes.

The diagram is the app's tier plan. Follow each task to its tier, and notice that the large tier still has a person behind it for big refunds.

Here is the plan in rupees, with the made-up prices from the table. Before tiering, all 11,400 daily requests went to the large model at Rs 0.90 each: Rs 10,260 a day.

After tiering, sorting costs Rs 60 a day, recipes Rs 1,200 and disputes Rs 360. Check it yourself: 3,000 × 0.02 + 8,000 × 0.15 + 400 × 0.90 = Rs 1,620 a day. The recipe screen also drops from 4.5 seconds to 1.8 seconds on average.

Tiering is decided once per task and does not look at each request. If requests inside a task are about equally hard, tiering is all you need. When difficulty varies a lot inside one task, you add a router or a cascade.

An LLM router is code in front of your models that looks at each request and sends it to one tier before any model answers. It can follow fixed rules you write, or use a small model to guess how hard the request is.

Rule-based routers

A rule-based router is plain if statements. It costs nothing, adds almost no wait and always does the same thing for the same input. The app's rules are:

• Requests from the recipe screen go to the medium tier.
• Requests from the refund dispute form go to the large tier.
• Free-text messages in the help chat need issue sorting, and no simple rule can tell how hard each one is.

Model-based routers

A model-based router asks a small, cheap model to read the request and predict whether the small tier can handle it. The team's first version did this for the help chat: "easy" went to the small tier, "hard" to the large tier.

Researchers have trained routers like this. A 2024 paper trained them on human preference data: which of two answers people liked more. It reported costs more than halved in some cases, without lower response quality.

Another 2024 paper routed by predicted query difficulty and a quality level you choose. It reported up to 40 percent fewer large-model calls with no drop in response quality. Both results are on the papers' own test sets, so measure on your traffic.

A router can be wrong, and it adds one extra small call to every request it reads. Test the router itself: in the made-up run, it sent 4 of the 20 hard sorting cases to the small tier, with no second chance.

The diagram shows that first version. Start at the incoming request, and see how the rules handle the known screens before the router model reads anything.

A cascade escalates a request when the first answer is not good enough. A fallback takes over when a model gives no answer at all. A cascade is about quality; a fallback is about errors, timeouts and outages.

How does a cascade work?

The small model always goes first, and your code checks its answer. If the check passes, you are done and paid only the small price. If it fails, the same request goes to the large model.

The check is the key part, and it must be cheap. For sorting, the check is that the answer is one of the five tags. The prompt also lets the model reply "unsure", which fails the check. A check like this, which looks only at the shape of the answer, is a format check.

A 2023 paper on cascades used a stronger check: a small scoring model trained to predict whether an answer is correct. It reported matching the best single model's performance with up to 98 percent lower cost on one of its test tasks, and smaller savings on the others.

The team replaced the router model in the help chat with a cascade. Follow one message through the diagram, and notice that a person is the last stop on both failure paths.

A cascade fits when most requests are easy and you have a cheap check. It fits badly when code cannot judge the answer, like a recipe that might be fine or might be bland. There, a fixed tier is the better choice.

What should a fallback do?

A fallback should do the safest useful thing when the planned model cannot answer. Hosted services do have errors, slow periods and outages, so plan for them from day one. The app has one fallback per task:

• Issue sorting: try the large model, and if that fails too, put the message in the support team's general queue.
• Recipe suggestions: show a saved list of popular recipes that use items in the cart. No model is needed.
• Refund disputes: send the dispute straight to a person, never to a smaller model.

Re-check whenever the model version, the prompt or your traffic changes, and on a fixed schedule such as monthly. Re-checking means running the same saved test set again and comparing the new scores with the old ones.

Hosted models can change under you. A 2023 study compared the March and June 2023 versions of the same hosted models. Behaviour changed a lot on some tasks, and the authors call for continuous monitoring.

If your provider offers fixed, dated model versions, pin one so updates do not reach you by surprise. Pinned versions can still be retired, so read your provider's deprecation page, which lists retirement dates. A model you host yourself keeps its learned parameters until you change them, but upgrading the software that runs it can still shift answers.

The team re-runs its test set when any of these happens. Each one can move a score:

• The provider announces a new version, or a retirement date for the pinned one.
• Anyone changes a prompt, even by one sentence.
• The share of sorting messages that escalate to the large model jumps, say from 12 in 100 to 25.
• A cheaper model appears that might pass a bar that used to need a bigger tier.

Each real mistake that support staff fix goes into the test set too. That way the test keeps matching what users actually send.

Less sure than the raw score suggests. A test set is a sample, so 94 of 100 is an estimate with a range around it, and that range can dip below your bar.

The code below uses the Wilson score interval, a standard 95 percent range for a pass rate. A range built this way contains the true rate about 95 times in 100. It also runs an exact paired test on the cases where only one of two models was right. It needs only the Python standard library.

from math import comb, sqrt

def wilson(passed, n, z=1.96):
    # 95% Wilson score interval for a pass rate
    p = passed / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half

def exact_paired_p(only_a, only_b):
    # Two-sided exact test on the cases where only one model was right
    n, k = only_a + only_b, min(only_a, only_b)
    tail = sum(comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

for passed, n in [(94, 100), (470, 500), (705, 750)]:
    low, high = wilson(passed, n)
    print(f"{passed} of {n}: {low:.1%} to {high:.1%}")
print(f"large right 4, small right 1: p = {exact_paired_p(4, 1):.3f}")

Running it on Python 3.13 prints:

94 of 100: 87.5% to 97.2%
470 of 500: 91.6% to 95.8%
705 of 750: 92.1% to 95.5%
large right 4, small right 1: p = 0.375

What just happened? With 100 cases, the true rate is plausibly anywhere from 87.5 to 97.2 percent, so 94 of 100 does not prove a 92 percent bar. With exactly 94 percent passing, 750 cases is the smallest test size whose whole range sits above 92.

The range also assumes your cases are a fair random sample of real traffic. The team added extra hard cases on purpose, so the score describes that harder mix, and real traffic will usually score a little higher.

The last line compares two models on the same cases. Say the large model was right on 4 cases where the small one failed, and the small one on 1 where the large one failed. Five cases decide the whole 3-point gap.

An exact test on those five gives p = 0.375. That means if the two models were equally good, a split at least this uneven would still happen about 37 times in 100. So the gap could easily be luck. The test looks only at cases where the two models disagree, which is McNemar's test in its exact form for small counts.

In practice you rarely need 750 cases for every task. The other two picks are less certain, because their tests are smaller. On recipes, 36 of 40 gives 76.9 to 96.0 percent, and on disputes, 55 of 60 gives 81.9 to 96.4 percent, both dipping below their bars. Grow the test set where it matters most: the tasks whose scores sit close to the bar, and the ones where a mistake is costly.

A cascade always pays for the small call, and pays for the large call on top for every escalated request. So it costs more than the small tier alone, and less than the large tier alone as long as most requests stop at the small model.

Per 1,000 requests, cost = small price + escalation share × large price. With the made-up prices and 12 in 100 escalated, that is 20 + 0.12 × 900 = Rs 128. Small alone costs Rs 20 and large alone Rs 900.

Wait time follows the same shape. The average is 0.5 + 0.12 × 4 = 0.98 seconds, but an escalated message waits about 4.5 seconds. With the timeouts in the code, the worst case is 3 + 8 = 11 seconds before the person queue.

One detail explains the numbers below. The cascade prompt lets the small model reply "unsure", so it answers only 88 of the 100 cases itself, with 86 right. The other 12 climb to the large model, which gets 10 right, 1 wrong and leaves 1 for a person. The small-only row uses the earlier prompt, which forces a tag.

Here is the cascade code. ask_model is a stand-in with fixed answers so the sample runs offline; replace it with your real model call. ModelUnavailable is the error your wrapper raises on a failure or timeout.

LABELS = {"missing_item", "late_delivery", "damaged", "wrong_item", "other"}

class ModelUnavailable(Exception):
    pass

# Stand-in answers so this sample runs offline. Swap in your real model call.
FAKE = {
    "small": {"Milk packet missing from order 7712": "missing_item",
              "Late again and the eggs are broken, refund now": "unsure",
              "This bag has someone else's things": "late_delivery"},
    "large": {"Late again and the eggs are broken, refund now": "damaged"},
}
DOWN = set()  # tiers that are failing right now

def ask_model(tier, message, timeout_s):
    if tier in DOWN:
        raise ModelUnavailable(f"{tier} gave no answer in {timeout_s} s")
    return FAKE[tier].get(message, "unsure")

def sort_issue(message):
    try:
        answer = ask_model("small", message, timeout_s=3)
        if answer in LABELS:
            return answer, "small"       # cheap answer accepted
    except ModelUnavailable:
        pass                             # small tier down: go up a tier
    try:
        answer = ask_model("large", message, timeout_s=8)
    except ModelUnavailable:
        return "needs_person", "fallback"  # no model could answer
    if answer in LABELS:
        return answer, "large"           # escalated answer accepted
    return "needs_person", "large unsure"

for message in FAKE["small"]:
    print(sort_issue(message))
DOWN.add("large")
print(sort_issue("Late again and the eggs are broken, refund now"))

Running it on Python 3.13 prints:

('missing_item', 'small')
('damaged', 'large')
('late_delivery', 'small')
('needs_person', 'fallback')

What just happened? The milk message stopped at the small model. The egg message got "unsure", escalated and came back as damaged from the large model. When the large tier was down, it went to a person.

Now look at the third line. The bag with someone else's things is a wrong item, but the small model said late delivery. That is a valid tag, so the check passed it and nothing escalated.

That is the blind spot. A format check only asks whether the answer is one of the five tags, so a wrong tag in the right format passes it. In the made-up test run, 2 of the cascade's 3 wrong tags came from the small model and passed this check.

So test the check itself: count wrong answers it accepted on your test set, as well as the escalation share. If that count is too high, use a stronger check, like the trained scoring model in the 2023 cascade paper, and re-test.

The team kept the cascade for sorting. It costs Rs 384 a day instead of Rs 60 for 3,000 messages, and sends the messages the small model is unsure about to the large one.

Treat model choice as a decision per task, backed by a test, not as picking one favourite model. Walk the interviewer from tasks to tiers, then to routing, failures and re-checks.

1. 1

   List the AI tasks

   Name each job the AI does, how often it runs and what a mistake costs. Here: issue sorting, recipes and refund disputes.

2. 2

   Set a pass mark for each task

   Say what a good answer means and what score is good enough, before you mention any model.

3. 3

   Test and pick tiers

   Describe a test set of real examples, and pick the cheapest tier that passes for each task. Mention the range around a score near the bar.

4. 4

   Route each request

   Use rules where the task is known, and a router model or a cascade where difficulty varies inside one task.

5. 5

   Plan for failure

   Give every task a safe fallback that passed its own test, and keep a person in the loop for costly decisions.

6. 6

   Re-check on change

   Re-run the test set on every model or prompt change, on a schedule, and when the escalation share moves.