Distillation: Making a Small Model Act Big
On this page
A ticket-routing feature runs on a large model and works well: it reads each support ticket and picks one of 40 queues. It also handles three million tickets a month, takes almost two seconds per call, and is one of the biggest lines on the AI bill. A small model prompted the same way gets the queue wrong far too often. Distillation uses the large model to teach the small one this one job, so the small model gets close to the large one's accuracy at a fraction of the cost and latency.
How it works
The large model is the teacher; the small one is the student. You run the teacher over many inputs that look like your real traffic and train the student to reproduce what the teacher does.

Distilling from answers (sequence-level)
- Collect a few thousand to a few hundred thousand real or realistic prompts.
- Have the teacher answer them, with your best prompt (few-shot examples, reasoning if it helps).
- Filter: drop wrong, low-confidence or malformed answers (check against labels where you have them, or with a judge).
- Fine-tune the student on the prompt → teacher answer pairs (ordinary SFT, often with LoRA).
This works with any teacher you can call, including a hosted API, as long as its terms of use allow training on its outputs. It is the most common form in product teams.
Distilling from probabilities (logit-level)
Instead of one answer, the teacher gives its full next-token distribution at every position. The student is trained to match it, usually with a KL-divergence loss, sometimes mixed with the normal loss on the true answer. A softmax temperature above 1 softens both distributions so the small probabilities, which carry "this is close but not quite" information, count more.
It transfers more information per example, but it needs access to the teacher's logits and the same tokenizer (or careful alignment), so it is mostly used when you control both models.
What the student learns
The student learns the teacher's behaviour on the distribution of inputs you showed it. It doesn't become a small copy of the teacher in general; outside that distribution it is still a small model.
Trade-offs and costs
- Narrow vs general: distillation works best for a well-defined task with a stable input distribution. Broad chat ability is much harder to distil well.
- Ceiling: the student rarely beats the teacher; teacher mistakes are copied unless filtered.
- Up-front cost: teacher calls to generate data, review, training, evaluation, and an ongoing pipeline to refresh when traffic changes.
- Ongoing savings: lower cost per request, lower latency, and the option to run the student on your own hardware or on device.
In production
- Generate from real traffic (with personal data removed) so the training inputs match what the student will see.
- Hold out an eval set labelled by people, not by the teacher, so you measure real accuracy, not agreement with the teacher.
- Route with a fallback: send traffic to the student, and escalate to the teacher when the student's confidence is low or the input looks unusual.
- Refresh the data set when products, categories or user behaviour change, and re-distil.
When to reach for it
- High volume of a narrow, repeated task (classification, extraction, routing, short structured answers).
- Latency targets the large model can't meet, or a need to run on your own GPUs or on device.
- The prompted small model is clearly not good enough, but the large model is.

Example
The ticket router again. The team takes 60,000 past tickets (personal data removed), has the large model label each with a queue and a confidence score, drops the 8% with low confidence, and fine-tunes an 8B open model with LoRA on the rest. On 1,000 tickets labelled by support leads, the teacher scores 92% and the student 89%. They route everything to the student and send its lowest-confidence 5% to the teacher, which brings combined accuracy to about the teacher's level. Latency drops from 1.8 s to 0.4 s and the monthly bill for the feature falls by more than 80%. (Numbers illustrative.)
Common mistakes
- Evaluating against teacher labels only, which measures imitation, not correctness.
- Training on synthetic prompts that don't look like real traffic.
- Ignoring terms of use of a hosted teacher model.
- Expecting a general small model to match a large one everywhere after task-specific distillation.
- No fallback for inputs the student has never seen.
Say this in the interview
"Distillation trains a small student model to reproduce a large teacher on our task. The usual version is sequence-level: run the teacher over real prompts, filter its answers, and fine-tune the student on them. If we control both models, logit-level distillation matches the teacher's full distribution with a KL loss and a temperature, which transfers more per example. It pays off for narrow, high-volume tasks. I'd evaluate on human-labelled data, not teacher labels, and keep the teacher as a fallback for low-confidence cases."
Interview questions on this
- Why do soft labels help? They show which wrong answers are nearly right, which a single hard label can't.
- What does the temperature do? Spreads out the probabilities so small ones contribute to the loss.
- Can the student beat the teacher? Occasionally on a narrow task with filtered data, but don't plan for it.
- How do you handle inputs the student wasn't trained on? Confidence-based routing to the teacher, and add those cases to the next training round.
- Distillation vs quantisation? Quantisation shrinks the same model's numbers; distillation trains a different, smaller model. They combine.
Used in
- AI Engineering: Fine-tune, RAG or Prompt?; Supervised Fine-tuning; Synthetic Data for LLMs; Model Routing and Fallbacks; How Would You Cut LLM Cost by 10x?
- AI System Design: Design a Fine-tuning Platform; Design an LLM Gateway.