Small Language Models: When Smaller Wins

You pay large-model prices and wait large-model times for jobs that need little of its skill, and the feature breaks whenever the network does. Small language models are the usual way out for narrow jobs like this.

A large language model, or LLM, is a very big text model, usually run in a data centre and reached over the internet. You pay your model provider for every request you send.

Picture a common setup. An expense app lets you point your phone at a shop receipt. It files the bill with the shop name, date, total and GST (the goods and services tax printed on Indian bills). The setup is made up, but the problem is real.

The first version is simple. The phone reads the printed text with OCR (optical character recognition, software that turns a photo of text into text). Then it sends that text to a large model in the cloud, which returns the four fields.

It works well in testing. Then usage grows to about 1.5 lakh (150,000) receipts a day, and the team lists three problems (all numbers made up).

• Cost. Each receipt costs about Rs 0.35 in model fees, so the app spends around Rs 52,500 a day on a job that looks the same every time.
• Waiting. A scan takes 3 to 6 seconds. Users in the billing queue give up and type the total by hand.
• No network. Many shops sit in crowded markets or basements with weak signal. There, the scan simply fails.

Users also ask a fair question: why does my pharmacy bill leave my phone at all? Receipts can reveal health purchases, home addresses and spending habits.

The team's answer is a small model that runs on the phone itself. This guide follows that change, where the small model beats the big one, and where the big one stays.

You need five words: parameter, inference, fine-tuning, distillation and quantization. Each gets a plain meaning and an example from the expense app.

Parameter. A parameter is one number inside the model that was set during training. A model's size is counted in parameters. The app's phone model has about 600 million, while large cloud models have many billions.

Inference. Inference means running a trained model to get an answer. Every receipt scan is one inference. Training happens once, but inference happens 1.5 lakh times a day, so its cost and speed matter most.

Fine-tuning. Fine-tuning means training an existing model a little more on examples of one task. The team shows the small model thousands of receipts with the correct fields, so it gets good at exactly that.

Distillation. Distillation means training a small model, the student, to copy the outputs of a larger model, the teacher. Here the large cloud model labels receipts, and the small model learns from those labels.

Quantization. Quantization means storing each parameter with fewer bits, so the model takes less memory and can run faster. Going from 16 bits to about 4 bits per number shrinks the file to a little over a quarter of its size.

Small language models are language models with few enough parameters to run cheaply on modest hardware, such as a phone, a laptop or one small server. They read and write text like large models, but usually hold less general knowledge.

There is no official cut-off for small. A 2024 survey of small language models says the meaning of small and large depends on context and time. A 2025 position paper defines a small model as one that fits on a common consumer device and answers one user quickly enough.

That same paper says its authors would call most models below 10 billion parameters small. Treat that as their working line, not a standard. What matters for you is whether the model fits your hardware and passes your tests.

Here is an everyday picture. Your sandal strap snaps near a busy market, and a roadside cobbler stitches it in five minutes for a small fee. You do not post the sandal to the brand's repair centre and wait two weeks.

• The cobbler is the small model: close by, quick and cheap for common fixes.
• The brand's repair centre is the large model in the cloud: it can rebuild a whole shoe, but it is far away and costs more.
• Posting the sandal is the network round trip: slow, and impossible when the post office is shut.
• A strap repair is a narrow, repeated task, such as reading a receipt total.

Where the picture breaks: a good cobbler knows when a job is beyond him and says so. You cannot count on a model, small or large, to tell you when it is wrong. That is why, later in this guide, your code makes the hand-over decision.

The 2024 survey makes the same practical point in its own words. It says small models suit settings such as on-device, mobile and edge use. Edge means running near the user rather than in a far data centre.

Here is the app's first version. Follow the arrows and notice that every single receipt crosses the network to the large model.

A small model beats a large one when the task is narrow, repeated and easy to check, and when speed, cost, privacy or offline use matter. It loses when the task needs broad knowledge or long chains of reasoning.

Reading a receipt is a good fit. The input always looks similar, the output is four fields, and code can check many of the answers. Nobody needs the model to know history or write poems to find a total.

Research backs this up for narrow tasks, within limits. In a 2023 paper on distilling step by step, a fine-tuned model of 770 million parameters beat a 540-billion-parameter model on one benchmark, a standard public test set.

The details matter. The small model used only 80 percent of that benchmark's training data, and it also learned from the large model's written reasoning. The large model was only prompted with a few examples, not fine-tuned.

A 2025 position paper goes further for AI agents, programs where a model picks steps in a loop. Its authors argue that small models are powerful enough and cheaper for many repeated agent calls. That is an argument, not a measurement, and they still suggest large models where open-ended chat is needed.

The team ran both models on 3,000 real receipts that people had checked by hand. Pause and predict before you read the table: which kind of receipt do you think the small model gets wrong most often?

Messy bills are the answer. The small model is slightly worse overall and clearly worse on faded or handwritten receipts. But it is faster, has no fee per scan and works offline, so the design must catch its weak cases.

You ask the small model for a fixed set of fields in a strict format such as JSON, then check its answer in code before you trust it. Anything that fails a check goes to a fallback.

Extraction means pulling named facts out of messy text. For the expense app, the facts are shop, date, total and GST. JSON is a simple text format of names and values, like a small labelled form.

A small model can skip a field, add a chatty sentence, or invent a total that is not on the bill. So the app checks every reply before saving it. The code below is the check alone, so you can run it without any model.

import json
import re

FIELDS = {"shop", "date", "total", "gst"}

def amounts_on_bill(ocr_text):
    # every amount printed with two decimals, such as 1,249.00
    found = re.findall(r"\d[\d,]*\.\d\d", ocr_text)
    return {float(a.replace(",", "")) for a in found}

def check_reply(reply, ocr_text):
    try:
        data = json.loads(reply)
    except json.JSONDecodeError:
        return "reject: not valid JSON"
    if not isinstance(data, dict) or set(data) != FIELDS:
        return "reject: missing or extra fields"
    total = data["total"]
    if not isinstance(total, (int, float)) or total <= 0:
        return "reject: total is not a positive number"
    if float(total) not in amounts_on_bill(ocr_text):
        return "reject: total is not printed on the bill"
    return "accept"

bill = "SAMPLE MEDICAL STORE\nDate 12/10/2026\nGST 59.48\nTOTAL 1,249.00"
replies = [
    '{"shop": "Sample Medical Store", "date": "2026-10-12", "total": 1249.00, "gst": 59.48}',
    'Sure! Here are the fields: {"shop": "Sample Medical Store"}',
    '{"shop": "Sample Medical Store", "date": "2026-10-12", "total": 1249.00}',
    '{"shop": "Sample Medical Store", "date": "2026-10-12", "total": 1294.00, "gst": 59.48}',
    '{"shop": "Sample Medical Store", "date": "2026-10-12", "total": -50, "gst": 0}',
    '{"shop": "Sample Medical Store", "date": "2026-10-12", "total": 1249.00, "gst": 5',
]
for reply in replies:
    print(check_reply(reply, bill))

Run it with Python 3. You should see these six lines, in this order.

• accept
• reject: not valid JSON
• reject: missing or extra fields
• reject: total is not printed on the bill
• reject: total is not a positive number
• reject: not valid JSON

What just happened? The first reply is clean, so it passes. The second wraps JSON in a chatty sentence, and the third forgets the GST field.

The fourth is the dangerous one: the total 1294.00 looks real, but the bill says 1,249.00. Only the last check, which compares the total with amounts printed on the bill, catches this kind of invented number.

The last reply was cut off mid-answer, which happens when a model hits its length limit. A model reads and writes text in tokens, small chunks of a word or a few characters, and runtimes stop after a set number of them.

The model does not decide; your code does. When a check fails, the code sends the receipt to a bigger model or a person.

Back to the cobbler, and the place where the picture breaks. A real cobbler would send a cracked sole to the brand centre himself, but a model will usually answer anyway. So the checks play the cobbler's judgement.

The diagram shows the new path. Start at the camera and find the question box. Notice that the cloud model is now one branch, not the main road.

Here is that flow in code. The helpers are stand-ins for the app's real code: small_extract runs the small model and returns its fields only if check_reply accepts them, otherwise None.

def read_receipt(photo):
    text = run_ocr(photo)                     # on the phone
    data = small_extract(text)                # small model + check_reply
    if data is not None:
        return save_expense(data, source="phone")
    if not has_network():
        return queue_for_later(text)          # retry when online
    data = cloud_extract(text)                # large model, text only
    if data is None:
        return ask_user_to_confirm(text)      # a person decides
    return save_expense(data, source="cloud")

Two lines matter most. The source= argument records which model produced each expense, so the team can measure both paths. And queue_for_later means a basement shop no longer breaks the scan; the receipt simply waits.

Now the money, with the made-up numbers. Say 88 percent of 1.5 lakh receipts pass on the phone. The other 18,000 go to the cloud at Rs 0.35, so the bill falls to about Rs 6,300 a day from Rs 52,500.

A small model can act as a router: it reads each request, gives it one label, and your code sends it down the matching path. Labelling is a narrow task, so a small model often does it well and cheaply.

The expense app also has a help chat. Users type things like "Where is my petrol bill?" or "Plan my monthly budget". The first is a simple lookup; the second needs reasoning across months of data.

Before, every chat message went to the large model. Now a small model on the app's own server labels each message first. The diagram shows the four labels and where each one goes.

Routers can also be trained. A 2024 research paper on routing between a strong and a weak model trained routers mainly on human preference data, where people voted for the better of two answers.

On the public benchmarks it tested, the paper reports cutting cost by over 2 times in certain cases, without losing response quality. Your saving depends on your own traffic mix, so measure it.

• Keep the label list short, around four to eight labels, each with clear examples.
• Always include an unsure label and send it to the safe path.
• Log every label, and have a person check a sample each week.
• Test the router on its own set of labelled messages before every change.

Choosing which model handles which kind of request is a wider topic, often called model tiering. A separate guide on choosing an LLM and routing between tiers covers it in depth.

You pick a small model, shrink it with quantization, and run it with an on-device runtime, a library built to run models on phone chips. Then you test memory, speed and battery on real, cheaper phones.

llama.cpp is one open-source engine that runs models on ordinary hardware. Its documentation includes an Android example app and a way to run it in a Linux-like terminal on Android.

Memory is the first wall. At 16 bits, each parameter takes 2 bytes, so a 1-billion-parameter model needs about 2 GB just for its numbers. On a phone, that memory is shared with the operating system and every other open app.

Quantization helps a lot. At 4 bits, each parameter takes half a byte, so the app's 600-million-parameter model needs about 300 MB. Real 4-bit files are a little larger, as the engineer section shows.

Quantization can lower accuracy, so the team reruns its 3,000 test receipts after every change. If you want the details of bits and formats, a separate guide on LLM quantization goes deeper.

• Download size. Ship the model as a separate download over Wi-Fi, not inside the main app, so installs stay small.
• Older phones. On phones without enough memory, skip the local model and use the cloud path, with the user's consent.
• Battery and heat. Longer replies mean more work for the chip, so ask the model for short JSON, never paragraphs.
• Updates. Ship a new model version like any app asset, and keep the old one until the new one passes a quick self-test.

You fine-tune it on many correct examples of that one task, and you can get those examples cheaply by distilling them from a larger model. Then you check the results on data the model never saw.

Distillation is an old idea. Earlier work showed that the knowledge in a group of models can be compressed into one model that is easier to deploy. A 2015 paper called this distilling and developed a different way to do it. Today the same idea is used to train small language models for one job.

1. 1

   Collect real inputs

   Gather 40,000 receipt texts from users who agreed to share them, with names and phone numbers removed first (made up).

2. 2

   Let the teacher label them

   The large cloud model extracts shop, date, total and GST from each one. These answers become training labels.

3. 3

   Check a sample by hand

   People check 2,000 labels and fix the errors. Faded bills get extra attention, because the teacher is weakest there too.

4. 4

   Fine-tune the student

   The small model trains on the labelled receipts. Keep 3,000 hand-checked receipts out of training, for testing only.

5. 5

   Shrink and test again

   Quantize the model to 4 bits and rerun the test set. Ship only if the total is right at least 95 percent of the time.

The test set is the most important part. It must never be used in training, or the model can look perfect on receipts it has already memorised.

You also do not always need to retrain every parameter. Some methods train only a small add-on and leave the model's own numbers unchanged, which is far cheaper. A separate guide on those adapter methods covers how they work.

Skip a small model when the task is open-ended, needs broad world knowledge, or changes shape every time. Also skip it when you have no way to check its answers or measure its mistakes.

• Open questions, like planning a budget or explaining tax rules, need wider knowledge and longer reasoning.
• Rare or new inputs, like a receipt in an unusual layout, fall outside what a narrow model was trained on.
• Low volume. At 200 requests a day, the cloud bill is tiny, and building a small model costs more than it saves.
• No test set. Without checked examples, you cannot tell whether the small model is good enough.
• Fast-changing rules. If the task changes every month, retraining the small model each time becomes a chore.

Small models also carry a quieter risk. Like large models, they can sound confident while being wrong. That is why every small-model path in the app ends in a check, never in blind trust.

More than half a byte per parameter, in the two llama.cpp formats this section measures. There, a "4-bit" model costs 4.5 bits per number or more, because each small block of numbers also stores its own scale.

A scale is the multiplier that turns the stored 4-bit codes back into real values. llama.cpp keeps models in GGUF files, its model file format, and names each storage format with codes like Q4_0 and Q8_0.

In llama.cpp's source, a Q4_0 block holds 32 numbers in 16 bytes plus one 2-byte scale. That is 18 bytes for 32 numbers, or 4.5 bits each. Q8_0 works the same way at 8.5 bits. A few other 4-bit formats, such as IQ4_XS, pack their blocks tighter, at 4.25 bits per number.

Real files come out larger still. llama.cpp's quantize documentation measures its common Q4_K_M preset, a named recipe for quantizing a whole model, at about 4.89 bits per number on one 8-billion-parameter model. Other models will land a little differently.

Pause and predict: how big is the app's 600-million-parameter model in each format? This short script does the sums; run it and compare.

params = 600_000_000
formats = [("16-bit", 16), ("Q8_0", 8.5), ("Q4_K_M", 4.89), ("Q4_0", 4.5)]
for name, bits in formats:
    size = params * bits / 8
    print(f"{name:7} {size / 1e6:6.1f} MB  {size / 2**30:.3f} GiB")

Watch the units when you check a real file. A MB here is 1,000,000 bytes, while a GiB is 1,073,741,824 bytes, so 337.5 MB shows as 0.314 GiB. File managers and phones do not all use the same unit.

At run time you need more than the file. The runtime also needs working memory for the text it handles, and that amount depends on the longest text you allow. So test peak memory on a real phone, not just the file size.

Fewer bits also means rougher numbers. In our own toy test, a random matrix stored as Q4_0 came back with about 8.6 percent relative error, against 0.5 percent for Q8_0. Relative error here is the size of all the errors together divided by the size of the original numbers. Real model quality does not map directly to that, which is why the team reruns its receipt tests.

No. A schema controls the shape of the reply while the model is writing, not whether the values are true. In llama-cpp-python 0.3.36, the latest release as of 19 October 2026, it also has gaps you should know.

llama-cpp-python is the Python wrapper around llama.cpp. You pass a schema through the response_format argument with type json_object, and the library turns it into a grammar, a set of rules that blocks any token breaking the shape.

What the schema cannot see or enforce

First, the model never sees the schema. llama.cpp's grammar documentation says the schema only constrains the output and is not added to the prompt, and llama-cpp-python's response_format code only builds the grammar. So describe the four fields in your prompt as well.

Second, the converter skips some rules. We converted the receipt schema with "minimum": 0 on the total, using the converter shipped in llama-cpp-python 0.3.36. The grammar it produced still allowed a minus sign, so a total of -50 fits.

llama.cpp's own documentation says minimum works only for whole numbers, and that unsupported features are skipped silently. In the Python converter we tested, minimum was dropped even for whole numbers. That is undocumented as of 19 October 2026; pin your version.

Third, a schema the converter cannot handle does not raise an error. In the 0.3.36 source, the chat code catches the failure and falls back to a plain JSON grammar that allows any keys. A schema the converter cannot read, such as an unknown type or a broken reference, quietly removes your field rules.

A cut-off reply is still possible

The grammar shapes each token, but generation still stops at the token limit. The library then reports a finish reason of "length", and the JSON is unfinished.

In 0.3.36, create_completion stops after 16 tokens unless you set max_tokens. create_chat_completion has no limit by default beyond the context size, the most tokens the model can handle at once, which you set when loading it. Set the limit on purpose, and treat a "length" finish as a failed check.

That is why the check_reply function from earlier stays, even with a schema. The schema makes rejects rarer and cheaper; the checks decide what you trust.

Explain them as a tool for narrow, high-volume tasks, placed behind checks and next to a larger fallback. Then back the choice with numbers.

1. 1

   Name the narrow tasks

   Point to the repeated jobs in the design, such as labelling messages or pulling fields from documents.

2. 2

   Say what matters most

   Is it cost per request, waiting time, privacy, or working offline? Pick the one or two that drive the choice.

3. 3

   Place the small model

   Say where it runs: on the device, on your own server, or both. For phones, mention memory, download size and real 4-bit file sizes.

4. 4

   Add checks and a fallback

   Describe the checks in code, including a cut-off reply, and the path to a larger model or a person when they fail.

5. 5

   Explain how it gets trained

   Fine-tuning on task examples, often distilled from a larger model, with a held-back test set.

6. 6

   Do the cost maths

   Estimate the share handled locally and what the fallback costs. Then say you would track that share daily.