Knowledge Distillation for LLMs: Small Models from Big Ones

The small model usually loses a lot of the large model's skill, and users notice. Yet you often need the small one, because a large model costs money per answer, needs a steady network and will not fit on a cheap phone. LLM distillation, where the large model teaches the small one, is one way out.

Picture a common setup. A team runs a Kannada voice assistant for budget phones. Farmers, shopkeepers and older users ask it questions out loud: bus timings, crop advice, or how to fill a pension form. The setup is made up, but the problem is real.

Today every question goes to a large model on rented cloud servers. It has 40 billion parameters, the numbers inside a model that it learned during training. It answers well, but each answer costs about Rs 0.40.

The app handles 2 lakh (200,000) questions a day, so that is Rs 80,000 a day. Worse, many users live where the network drops often. Reviews keep saying the app spins and then fails.

So the team wants the assistant to run on the phone itself. Their target phones have 4 GB of memory, and the app can safely use about 1 GB of it. That is where the maths stops them.

Models are often served in a 16-bit format, which takes 2 bytes per parameter. At that size the big model needs 80 GB just to hold its parameters. Even a model with 1 billion parameters needs 2 GB.

Storing each parameter in fewer bits can come later. First the team needs a small model that is actually good at Kannada help questions. So they try the easy route: fine-tuning, which means training an existing model a little more on your own examples.

They fine-tune a 1 billion parameter model on 50,000 question and answer pairs checked by Kannada speakers. They also keep a test set: 2,000 real questions with checked answers that never go into training.

The big model gets 90% of the test set right. The small fine-tuned model gets 71%. Users would notice that gap at once. The next idea is distillation: let the big model teach the small one.

You need five words: teacher, student, logits, soft labels and loss. Each one gets a plain meaning and an example from the voice assistant.

Teacher. The teacher is the large, accurate model you already have. Example: the team's 40 billion parameter model, an open-weight model (one whose parameters you can download and run) hosted on its own cloud servers.

Student. The student is the small model you want to ship. Example: a 1 billion parameter model that must fit on a 4 GB phone.

Logits. A language model writes one token at a time. A token is a small piece of text, often a word or part of a word. For every possible next token, the model gives a raw score called a logit, and a higher logit means more likely.

Soft labels. Turn the logits into probabilities that add up to 1, and you get the model's full guess for the next token. Distillation calls these soft labels. A hard label is just the one correct token, with nothing said about the rest.

Loss. Loss is a number that says how far the model's output is from what you want. Training nudges the student's parameters to make the loss smaller, over millions of small steps.

LLM distillation is a way to train a small language model to imitate a large one, using the large model's outputs as the training signal. The goal is a student that keeps most of the teacher's skill on your task, at a fraction of the cost.

The idea is older than large language models. Rich Caruana and colleagues had already shown that a group of many models could be compressed into one model that is much easier to deploy. In 2015, Hinton, Vinyals and Dean built on that work, and their name for it, distillation, stuck.

Picture a qawwali party, an evening of Sufi devotional singing, on a Thursday night. The lead singer carries the song, and young chorus singers sit behind him and sing along. Night after night, they copy how he stretches a note and where he takes a breath.

Map it part by part. The lead singer is the teacher and the chorus singers are students. The lyrics are the hard labels, and the way he sings them is the soft labels. A youngster who only memorised the lyrics would sing the right words, but flat.

Where the picture breaks: a chorus singer hears only the one note the lead actually sang. Soft labels give more than any listener hears, because they hold the teacher's odds for every note it could have sung.

For the voice assistant, the 50,000 checked pairs are the lyrics. The teacher's probabilities on every token are the phrasing. The team wants the student to learn both.

Here is the basic setup. Follow one question from the top. Notice that the teacher is frozen: it only answers, and only the student changes.

Soft labels tell the student which wrong answers are close and which are absurd, and a plain correct answer cannot. That extra information lets a small model learn more from every example.

Take a user who asks for the first bus from Mysuru to Bengaluru. The reply begins "The first bus leaves at", and the model must pick the next token. The app answers in Kannada, but this guide shows the words in English.

Say the teacher's probabilities for four possible next tokens are these: 6 gets 0.67, 7 gets 0.30, morning gets 0.03, and banana gets 0.002. Rounded like this they come to just over 1; unrounded they add up to exactly 1.

The hard label says only 6. The soft label says more: 7 was also a fair guess, a time-of-day word was possible, and banana makes no sense. The student learns how the teacher weighs options, not just what it picked.

The 2015 paper makes the same point with an image example. A photo of a BMW is rarely mistaken for a garbage truck, but that mistake is still many times more likely than calling it a carrot. These small differences show how the model generalises, meaning how it handles inputs it has never seen.

With one fixed correction, nearly all of them. The authors raised the student's starting score for the digit 3 by a constant amount, picked using the test set itself. After that, it got 98.6% of the test 3s right, learned purely from the teacher's soft labels on other digits.

What temperature does in distillation

Temperature is a number that softens the probabilities before the student copies them. You divide every logit by the temperature, and a higher value spreads probability more evenly across tokens.

At temperature 1, the teacher above gives 6 about 67% and banana 0.2%. At temperature 2, the same logits give 6 about 51%, 7 about 35%, morning about 11% and banana about 3%. The quiet signals become loud enough to learn from.

In the 2015 recipe, teacher and student use the same high temperature during distillation. Once training ends, the student goes back to temperature 1 for real answers.

If you code in Python, here is the classic distillation loss for one token, using NumPy, a common library for maths on lists of numbers. The softmax function turns logits into probabilities at a chosen temperature. If you do not code, skip to the result below it.

import numpy as np

def softmax(logits, temperature=1.0):
    scaled = np.array(logits) / temperature
    exps = np.exp(scaled - scaled.max())
    return exps / exps.sum()

def distill_loss(teacher_logits, student_logits, true_index, t=2.0, alpha=0.5):
    p_teacher = softmax(teacher_logits, t)
    p_student = softmax(student_logits, t)
    soft = np.sum(p_teacher * np.log(p_teacher / p_student)) * t * t
    hard = -np.log(softmax(student_logits)[true_index])
    return alpha * soft + (1 - alpha) * hard, soft, hard

# tokens: 6, 7, morning, banana
teacher = [4.0, 3.2, 1.0, -2.0]
student = [3.0, 1.0, 1.5, 0.0]
print(np.round(softmax(teacher, 2.0), 3))
total, soft, hard = distill_loss(teacher, student, true_index=0)
print(round(total, 3), round(soft, 3), round(hard, 3))
print(round(distill_loss(teacher, teacher, 0)[1], 6))

Run it with NumPy installed (we used NumPy 2.5.4). It prints three lines: [0.515 0.345 0.115 0.026], then 0.456 0.57 0.342, then 0.0.

The soft part measures how far the student's softened probabilities are from the teacher's. This score is called KL divergence, a standard way to compare two probability lists. It is zero when they match.

The t * t factor comes from the 2015 paper. Gradients are the signals that tell training which way to nudge each parameter. At high temperatures the soft part's gradients shrink roughly by the square of the temperature, so multiplying back keeps both parts balanced.

The hard part is ordinary training on the correct token, at temperature 1. The alpha value sets the mix between the two parts. The 2015 paper got its best results with a much lower weight on the hard part, so treat alpha as a setting to tune.

There are four main types: copy the teacher's probabilities, copy its written answers, copy its reasoning, or let the student answer while the teacher scores it. They differ mostly in how much access to the teacher you need. They also overlap: on-policy distillation, as described below, needs the teacher's token probabilities, and rationale distillation is black-box distillation with reasoning added.

White-box distillation: copy the probabilities

White-box distillation uses the teacher's internal outputs, such as its probabilities over every token. In practice that means running the teacher yourself. A 2024 survey of LLM distillation by Xu and colleagues says it is not suitable when the teacher's internal parameters are inaccessible.

There is a second catch. The teacher's probability list and the student's must cover the same tokens, so both models need the same vocabulary. A student built on a different tokenizer, the tool that splits text into tokens, cannot be compared token by token without extra alignment work.

Black-box distillation: copy the written answers

Black-box distillation uses only the text the teacher writes. You send it many questions, save its answers, and fine-tune the student on them like ordinary labelled data. It works even when you can only reach the teacher through an API, a way for programs to call each other.

Older research has a name for this. In 2016, Kim and Rush called it sequence-level distillation and tested it on translation models. The student trained on the teacher's best-scoring full translations instead of the human ones.

Their best student, translating word by word with greedy decoding, ran 10 times faster than the teacher with beam search, with little loss in quality. That speed was measured on a GPU, the graphics chip most models run on. Beam search keeps several candidate translations and returns the best-scoring one, which is slower.

Rationale distillation: copy the reasoning

Here the teacher writes its reasoning steps as well as the answer. In the 2023 distilling step-by-step method by Hsieh and colleagues, the student trains on two separate tasks: give the answer, and write the reasoning. At test time it only needs to give the answer.

Their result is striking. On one benchmark, a standard test set, a 770 million parameter student beat its own 540 billion parameter teacher, which was prompted with a few worked examples. The student used only 80% of the available training data.

On-policy distillation: let the student answer alone

In on-policy distillation the student writes its own answers, and the teacher scores every token of them. It needs the teacher running during training, and it gets its own section below.

On-policy distillation trains the student on answers it wrote itself, with the teacher scoring each token. It targets a gap in normal distillation, where the student only ever practises on fixed text, never on its own words.

Agarwal and colleagues described this gap in a 2023 paper. In training, the student sees fixed text, written by the teacher or by people. In real use, it continues from its own words, mistakes included.

Their method, generalized knowledge distillation (GKD), trains the student on its own outputs with the teacher's token-by-token feedback. A setting called lambda picks what share of training answers the student writes itself. The rest come from a fixed dataset.

The team saw exactly this gap. In long answers, such as the steps for a pension form, the student made one small slip early. It had never practised recovering, so the rest of the answer drifted further off.

Back at the qawwali party, singing in the chorus is easy, because the lead singer's voice keeps you on pitch. The hard part comes when he nods at a young singer to take a verse alone. If the youngster drifts, the lead singer picks up the line right there.

The youngster learns to recover from his own slip, which chorus singing never teaches. The diagram shows the same loop. Notice that the teacher only scores; it never writes the answer.

The key step is the teacher call. The teacher reads the student's draft, not its own answer, and says what it would predict at each position. The paper treats the draft as fixed text and does not send gradients back through sampling, the step where the student picks each token at random by its probability. The paper says this keeps training stable and efficient.

The paper also lets you choose the divergence, the formula that scores the gap. That helps when the student is too small to copy the teacher fully, a case the engineer section below tests.

This costs more. The student must write drafts during training, which is slow, and the teacher must score every draft. The team uses it only for a final round, on long answers where drift hurt most.

You build it in six steps: collect real questions, get teacher outputs, filter them, train the student, shrink and test it on phones, then ship with a fallback. Most of the effort goes into the data, not the training code.

1. 1

   Collect real questions

   Pull 3 lakh past user questions from the logs, with names and phone numbers removed. Real questions carry real dialects and real mistakes that invented ones miss.

2. 2

   Get teacher outputs

   The teacher answers every question. For white-box training, also save its top 20 token probabilities at each position.

3. 3

   Filter the outputs

   Code drops answers in the wrong language, answers that run too long, and anything the safety check flags. A wrong teacher answer becomes a wrong lesson for the student.

4. 4

   Train the student

   Train with the mixed loss: soft labels from the teacher, plus hard labels from the 50,000 checked pairs. A final on-policy round fixes drift on long answers.

5. 5

   Shrink and test on real phones

   Store each parameter in 4 bits instead of 16, which brings the student's parameters to about 0.5 GB. Then test speed and battery use on three budget phones.

6. 6

   Ship with a fallback

   The student answers on the phone. When it is unsure, or needs fresh data like today's crop prices, the app asks the cloud teacher if the network allows.

Check the shrink step yourself: 1 billion parameters times 4 bits, divided by 8 bits per byte, is 500,000,000 bytes. Many tools show that as 0.466 GiB, because a GiB is 1,073,741,824 bytes. The phone also needs working memory on top of that.

The diagram shows the same pipeline as parts of a system. Notice that the cloud teacher appears twice: once to teach, and later as a fallback.

A distilled model is good enough when it scores well on your own test set, runs on the real phone, and stays within your speed and battery limits. The teacher's score is a guide, not a limit: on narrow tasks a student can sometimes beat it.

Here is where the team landed on the same 2,000 test questions. Every number here is made up.

• Teacher on the cloud: 90% correct, about 1.5 seconds per answer when the network is good.
• Small model fine-tuned on checked pairs only: 71% correct.
• Distilled student: 84% correct, about 0.8 seconds per answer on the phone, with no network needed.
• After the on-policy round, long form-filling answers rose from 79% to 86% correct.

Averages hide things, so split the results by question type and by region. Kannada spoken in Mangaluru differs from Kannada spoken in Kalaburagi. In this setup the student was weakest on north Karnataka phrasing, so the team added more of those questions to the training data.

Add a product check too. Give 50 users the offline version for a week and compare their complaints with users on the cloud version. A test set tells you about answers; real users tell you about the whole experience.

Use distillation when you need a smaller model that keeps a specific skill and you have a stronger model to learn from. Use quantization or pruning when you only need the same model to take less memory.

Quantization stores each parameter with fewer bits, such as 4 instead of 16. Pruning removes parameters that matter little. Both shrink one existing model; neither teaches a new one.

They combine well. Kim and Rush pruned 80% of the parameters of a distilled student and got 13 times fewer parameters than the teacher. Its translation quality was only 0.4 BLEU below the teacher's, where BLEU is a standard translation score. The voice assistant team does the same with distillation, then quantization.

Distillation also differs from plain fine-tuning, though it uses the same training machinery. Fine-tuning learns from the labels you have. Distillation makes far richer labels from a teacher. Our guide on how LLMs are trained covers the training stages underneath both.

Run a quick check before you commit to it. Distillation is worth the effort when most of these hold:

• You have a strong teacher and a narrow task, like Kannada help questions.
• The model must run on cheap hardware or without a network.
• You serve so many requests that a cheaper model pays back the training cost.
• Your team can spend a few weeks on data work and testing.

Skip it when your task changes every few weeks, when a ready-made small model already scores well enough, or when you have no test set to prove the student works. In those cases, plain fine-tuning or the ready-made model costs less.

Distillation can copy the teacher's mistakes, lose rare skills, break licence terms, or produce a student that looks fine on average but fails on edge cases. Each problem has a guard.

• Copying the teacher's mistakes. The student learns whatever the teacher says, wrong facts included. Guard: filter teacher outputs and keep human-checked pairs in the mix.
• Losing rare skills. A small model has less room, so rare topics may suffer first. Guard: test rare question types separately and send weak ones to the cloud fallback.
• Sounding too sure. A student can sound as confident as the teacher while knowing less. Guard: hand off when its token probabilities are low, after checking on your test set that low really means wrong.
• Stale facts. The student knows only what was in its training data. Guard: fetch prices and timings from a live source, never from the model's memory.
• Too big a gap. In a 2019 study on image models, Mirzadeh and colleagues found a teacher transfers well only to students down to a certain size. Guard: add a middle-sized teacher assistant model, or pick a bigger student.
• Ignoring licence terms. A teacher's licence or terms of service may limit using its outputs to train another model. Guard: read them before generating a single example.

Privacy needs the same care. The questions in step 1 came from real users, and anything left in them can end up inside the student. Remove names, phone numbers and Aadhaar-like digit strings before the teacher sees anything.

Use forward KL when you want the student to cover everything the teacher thinks possible. Use reverse KL when you want it to commit to the teacher's most likely answers. The choice matters most when the student is too small to copy the teacher exactly.

Forward KL, written KL(teacher || student), scores the gap from the teacher's side. It punishes the student hardest for giving low probability to tokens the teacher likes. Reverse KL, written KL(student || teacher), scores it from the student's side, so it punishes the student for putting probability where the teacher puts little.

We tested this with a toy: a teacher with two equal bumps, and a student that can only make one bell curve (a Gaussian). We searched for the best student under each loss on NumPy 2.5.4.

• Forward KL put the student's bump halfway between the teacher's two, wide enough to touch both. It placed 24% of its probability in the gap, where the teacher has 0.6%.
• Reverse KL sat the student exactly on one of the teacher's bumps and ignored the other. Its probability in the gap matched the teacher's 0.6%.

In token terms, a forward KL student may say things the teacher would rarely say. A reverse KL student sticks to answers the teacher likes, but can lose variety. The toy is a picture of the effect, not a measurement on a language model.

The papers pull in both directions. A 2023 paper by Gu and colleagues switched to reverse KL and reported better answer quality and calibration, meaning its confidence better matched how often it was right. The GKD paper found the best divergence is task-dependent, so try both on your own test set.

Far more than the answers themselves, unless you keep only the top few tokens per position. Saving every probability at every position quickly runs into terabytes.

Take round numbers for the voice assistant: 3 lakh answers, 150 tokens each, and a vocabulary of 100,000 tokens. These sizes are made up for the maths. At 2 bytes per probability, the full lists come to 9 TB.

Keep only the top 20 tokens per position, each with a 4-byte token id and a 2-byte probability. That is 5.4 GB, about 1,667 times smaller. The arithmetic is 300,000 times 150 times 20 times 6 bytes.

The saving has a cost. The student never sees the tail of the teacher's list, so you must decide what to do with the missing probability. You can rescale the kept ones to add up to 1, or keep the leftover as one extra everything-else entry.

On-policy rounds cannot use saved probabilities at all. The student's drafts do not exist until training runs, so the teacher must score them live.

The open-source TRL training library ships it. As of 19 October 2026, TRL 1.16.0, released on 15 October 2026, has one stable trainer for it and several experimental ones. We read the stable one and the experimental GKDTrainer in the released source code, without running them.

• DistillationTrainer, in the stable API, trains fully on-policy. Its beta setting picks the divergence: 0 is forward KL, 1 is reverse KL, and the default is 1.0.
• GKDTrainer sits under trl.experimental, whose import warning says its APIs may change or be removed without notice. It mixes in fixed data, with lmbda (TRL's spelling of lambda) 0.5 and beta 0.5 by default.
• The beta setting means the same in both, but the defaults differ. Copying a GKD config into DistillationTrainer without setting beta changes your loss.

When you pass a teacher model, both trainers check that it and the student report the same vocabulary size, and stop with an error if not. The error points to GOLD, another experimental trainer, for teacher and student pairs with different tokenizers.

To confirm which version you have, run pip show trl in your environment. The Version line should read 1.16.0 if you pinned the release described here.

Frame it as a cost and latency decision first. Then walk through the teacher, the data, the training type, the evaluation and the fallback, in that order.

1. 1

   State the constraint

   Say why a smaller model is needed: phone memory, offline use, cost per request or latency, which means how long one answer takes.

2. 2

   Pick teacher and student

   Give rough sizes, say whether you can run the teacher yourself or only call it, and check the two share a vocabulary.

3. 3

   Choose the type

   White-box if you have the teacher's probabilities, black-box if you only have its text, and an on-policy round for long answers.

4. 4

   Design the data pipeline

   Real questions, teacher outputs, filters, and a human-checked test set that never goes into training.

5. 5

   Define done

   A target score on your own test set, split by question type and region, plus speed and battery limits.

6. 6

   Plan the fallback

   Send low-confidence or fresh-data questions to the bigger model when the network allows.