Model Distillation Explained: How AI Builds Faster, Smaller Models

AI models keep getting bigger. GPT-4 reportedly has over a trillion parameters. Google’s Gemini Ultra is trained on more data than most humans will ever read in a lifetime. But here’s the uncomfortable truth — bigger models are slow, expensive to run, and out of reach for most developers. Model distillation is the fix. It’s how AI labs take a huge, brilliant model and squeeze its knowledge into something that runs on your laptop or phone. The results are genuinely impressive, and the technique is reshaping how AI gets deployed at scale.

When I first looked into distillation, I expected a crude compression trick — something that trades accuracy for speed. It’s actually far more nuanced than that. Done well, a distilled model can match 85–95% of its teacher’s performance while running ten times faster and costing a fraction to operate. That gap closes fast, and in many real-world tasks, the distilled version wins outright.

What Is Model Distillation?

Model distillation is a training technique where a smaller “student” model learns from a larger “teacher” model. The concept was formalised in a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean at Google. The insight: raw training labels (correct/incorrect) contain less information than the full probability distribution a teacher model outputs.

When a teacher model looks at a photo of a cat, it doesn’t just say “cat.” It might say: 92% cat, 6% fox, 1% dog, 0.5% rabbit. Those soft probabilities carry a lot of implicit knowledge — what makes a cat similar to a fox, where the model is uncertain, which features matter. A student trained on those soft labels learns faster and generalises better than one trained on hard labels alone.

Think of it like this. You could learn French by memorising a dictionary. Or you could learn from a fluent speaker who explains grammar rules, gives examples, and corrects your nuances. The second approach transfers knowledge, not just facts. That’s distillation.

How the Training Process Actually Works

In a standard distillation setup, the teacher model processes inputs and generates a probability distribution — called soft targets or logits. The student model trains on a combined loss: part from the original hard labels, part from matching the teacher’s soft targets. The temperature parameter controls how “soft” those targets are. A temperature of 1 gives the raw probabilities. Higher temperatures spread them out, making small differences more visible to the student.

The training runs in two stages. First, you train the teacher as normal — no distillation involved. Then you freeze the teacher and train the student, feeding it the teacher’s outputs as supervision. The student is typically 10x to 100x smaller. A teacher with 70 billion parameters might produce a student with 7 billion.

Some labs use “online” distillation where teacher and student train simultaneously, or “self-distillation” where a model teaches itself across iterations. As of 2025, the most sophisticated implementations use layer-by-layer matching, where the student tries to mimic not just the final output but the internal representations at each layer.

Real-World Examples That Changed the Game

DistilBERT, released by Hugging Face in 2019, was one of the first widely-used distilled models. It reduced BERT’s size by 40%, sped it up by 60%, and retained 97% of BERT’s language understanding performance on most benchmarks. Suddenly, companies could run a BERT-class model on commodity hardware.

Meta’s Llama models have driven much of the distillation conversation since 2023. The Llama 3.1 8B model — distilled from a much larger teacher — performs comparably to GPT-3.5 on most benchmarks. That’s a model you can run locally on a decent laptop. Google’s Gemma 2 series used distillation to punch far above its weight class. Gemma 2 9B regularly beats models three times its size on reasoning tests.

The most striking recent example is DeepSeek-R1. The Chinese lab released distilled versions trained from their R1 reasoning model — a 7B distilled version that rivals Claude 3.5 Sonnet on several coding benchmarks. When that news landed in early 2025, it shook assumptions about what small models could do.

Why the “Small Model” Revolution Matters

Cost. That’s the blunt version. Running GPT-4 via API costs roughly £0.015 per 1,000 tokens. A well-distilled open model running on your own hardware can reduce that to near zero. For a startup processing millions of user queries daily, this is the difference between viable and ruinous unit economics.

Privacy matters too. When you query a cloud model, your data leaves your network. A distilled model running locally keeps everything on-device. In regulated industries — healthcare, legal, finance — that’s not optional. The NHS cannot route patient data through a US API. A distilled model running on a hospital server can.

Speed is the third driver. API latency for large models often sits at 1–5 seconds for short responses. A distilled model on a local GPU can return responses in under 200 milliseconds. That’s real-time territory. Voice assistants, autocomplete systems, and interactive tools all need sub-second response times — something large cloud models fundamentally cannot deliver.

The Limits: What Distillation Can’t Do

Distillation isn’t magic. The student is bounded by the teacher. You can’t distil capabilities the teacher doesn’t have. If the teacher hallucinates on niche medical questions, the student will likely hallucinate too — possibly more, because it has less capacity to store edge-case knowledge.

There’s also a task-specificity problem. Distilled models often shine on the tasks they were trained for and degrade badly on others. A model distilled for customer service chat might underperform on complex code generation. General-purpose distillation is significantly harder than domain-specific distillation.

The gap widens for the hardest tasks. Mathematical olympiad problems, multi-step logical reasoning over long documents, complex code debugging across large codebases — these still require the full scale of frontier models. The state of the art in distillation as of 2026 is roughly: 90% of performance at 10% of the cost for typical tasks, dropping to 70% of performance for the hardest 5% of tasks.

How UK Developers Are Using Distillation

When I’ve spoken to UK developers building AI products, distillation comes up constantly — usually framed around the make-or-buy decision. The choice is: pay OpenAI or Anthropic per token, or run your own distilled model. The crossover point depends on volume. Below roughly 500,000 queries per month, cloud APIs usually win on cost. Above that, self-hosted distilled models start to make economic sense.

Several UK fintech companies are using Llama-based distilled models for document summarisation and compliance checking. The models run on servers they already control, which satisfies FCA data residency concerns without expensive cloud contracts. Startups building coding assistants have started fine-tuning distilled models on their own codebases — you get a model that knows your codebase’s conventions better than a generic cloud model ever could.

The practical barrier has dropped dramatically. In 2023, running a 7B parameter model required a high-end GPU. In 2026, it runs reasonably well on an M-series Mac. Tools like Ollama and LM Studio have made local model deployment accessible to developers who’ve never touched GPU infrastructure.

Quantisation vs Distillation: Not the Same Thing

These terms get confused, and it matters. Distillation creates a new, smaller model trained from a larger one. Quantisation takes an existing model and reduces the numerical precision of its weights — typically from 32-bit floats to 8-bit or 4-bit integers. Quantisation doesn’t change model architecture, it just shrinks file size and memory requirements.

Both techniques are often used together. You distil a 70B model down to 7B, then quantise the 7B model to 4-bit to fit in 4GB of RAM. The cumulative effect is dramatic: a distilled, quantised model can be 50x smaller than the original while maintaining surprising capability.

Pruning is a third related technique — removing weights close to zero from an existing model on the assumption they contribute little. The research consensus as of 2026 is that distillation usually outperforms pruning for large compression ratios, but pruning is faster to apply to an existing model without additional training.

The Future: Speculative Decoding and Beyond

Distillation is evolving fast. One of the most promising current directions is speculative decoding, where a small distilled “draft” model generates candidate tokens, and a large model verifies or corrects them. This hybrid approach gets close to small-model speed with large-model accuracy. Google uses a variant of this in its Gemini deployment.

Mixture-of-Experts architectures are also changing the distillation calculus. Rather than one large dense model, MoE models activate only a subset of their parameters per token. Mistral’s Mixtral 8x7B effectively operates like a 7B model per token despite having 47B total parameters. Distilling from MoE teachers into dense students is an active area of research in 2026.

The 12-month direction is clear: the capability gap between frontier models and distilled models will continue to narrow. The tasks where you genuinely need a trillion-parameter model are shrinking. For most practical applications — writing assistance, document processing, code completion, customer service — distilled models are already good enough, significantly cheaper, and meaningfully faster.

What This Means for UK Developers and Businesses

If you’re building anything AI-powered, understanding distillation changes your architecture decisions. It’s not about ditching cloud models entirely — it’s about knowing when a distilled local model serves the job better and when frontier capability is genuinely required. Most applications don’t need the full power of GPT-4. They need something fast, private, and cheap that gets the job done 90% as well. That’s distillation’s sweet spot.

For UK businesses navigating GDPR, data residency requirements, and FCA oversight, the case for local distilled models is strong wherever sensitive data is involved. The tooling has matured enough that deployment is no longer an infrastructure nightmare. If you haven’t evaluated distilled models for your use case in the last six months, the landscape has changed enough that it’s worth another look.

This article is for educational purposes only and does not constitute financial or investment advice. Always do your own research before making technology or business decisions.