Distillation: Racing to the frontier
How AI model distillation lets smaller models learn from frontier systems, what gets lost through an API, and why the frontier keeps moving.
Alex Smyntyna
July 29, 2026 · 7 min read
Two things with one name
Distillation is teaching a small model using a larger one. That's the whole idea, and it covers two completely different techniques.
The difference is access. Either the teacher is a file on your disk that you can open, or it's an api endpoint you send requests to. With the file you can copy the teacher's insides. With the endpoint you get text back and nothing else.
Both setups train a small student to imitate a big teacher. What differs is how much of the teacher is readable while it happens.
Inside the lab
The trainer holds the weights. Every step can read the teacher from the inside, and the loss compares two distributions directly.
requests billed 0 — the file is already open
Through the API
The trainer sends prompts and receives text. The model itself stays behind the wall, and the student learns from transcripts.
requests billed 1,247 — one sampled token each
- Carries
- The tokens the teacher actually emitted, one sequence out of the very many it could have written.
- Through the API
- The only channel that survives the trip, and the thinnest one. A single sample stands in for a whole distribution.
- The student sees, per token
- All 128,256 probabilities, plus the hidden states and attention at every layer.
- The loss can compare
- Two distributions directly, token by token. KL divergence on soft targets.
- Out of reach
- Nothing structural. Only compute, data, and how much the student can hold.
- The student sees, per token
- One sampled token. At most a few top log probabilities, if the vendor exposes them.
- The loss can compare
- The student’s guess against the one token the teacher happened to emit. Cross entropy on a transcript.
- Out of reach
- The distribution, hidden states, attention, gradients, and everything the teacher nearly said instead.
The best-known open example of the second kind is a reasoning model that was used to generate roughly 800,000 sample answers, which were then used to fine-tune several smaller open models. No reinforcement learning and no weights. Just the text.
Both of these get called distillation. Nearly every model small enough to run on your own machine is one or the other.
What the model has before it answers
The model never picks a word. It scores every word it knows.
A current open model's vocabulary is around 128,000 tokens. Every time it writes one word, it produces 128,000 numbers, one for each. Then one gets chosen and the rest are thrown away.
The thrown-away part is the interesting part.
What the student actually receives
The same teacher, the same token position. The only thing that changes is whether you hold the weights or hold an API key.
prompt
The committee reviewed the conflicting evidence for hours, but eventually agreed that the proposal was { 128,256 probabilities }
Switch to API only to run the experiment
The full distribution is already visible here. The API tab lets you discover how slowly sampled text reconstructs it.
- sound0.900
- flawed0.050
- viable0.020
- impractical0.010
- promising0.00600
- incomplete0.00400
- risky0.00300
- reasonable0.00200
- premature0.00100
- defensible0.00080
- ambiguous0.00070
- misguided0.00050
- tail128,244 other tokens0.00200
The student sees this whole vector, for every token of every training example. Not the answer, the shape of it: flawed is live, viable beats impractical, misguided is worse than all of them and still an answer.
internal: what crosses the wire
128,256 real numbers, per token
A full real-valued vector over the vocabulary. Ranking, margins and near-misses all survive the trip.
external: what crosses the wire
1 token, at most 17.0 bits
One choice out of 128,256 identifies which option won, and nothing about by how much.
This is distillation when you own the teacher. The soft target is the training signal, and it carries far more than the one correct answer a normal dataset gives you.
Say sound comes out on top at 0.90. flawed is sitting at 0.05 and viable at 0.02. That spread carries a judgment the final word cannot: the teacher is confident, but it can still see the criticism and the plausible alternative. None of that survives into the text.
Hinton's 2015 paper named this. He called it dark knowledge: what the model believes about the answers it didn't give. It's what you copy if you have the file, and what you lose if you don't.
What it costs to guess
The obvious objection is that you could just ask for a lot of samples and work the shape out from those. That's true. Here's the price.
To learn a word that comes up 1% of the time, accurate to within 10%, takes about 9,900 samples. To learn the 90% winner takes 12.
Those two numbers are here because the gap between them is the whole problem. Common things are almost free to learn. Rare things cost eight hundred times more, and rare things are most of a vocabulary.
Then there's the part that ends the argument. Real training data sees each sentence once. Not 9,900 times. Once. So the shape never gets measured at all, you get a single draw and move on.
It still works, because millions of different sentences average the noise out. It works well enough that most open models are built this way. But nobody who owns the weights chooses it.
Then models started showing their work
Something changed in the last couple of years and nobody planned it.
Reasoning models write their steps out before answering. Which approach they picked, what they worked out halfway through, where they went wrong and backed up. All of it arrives as text, through the same api that used to hand back a bare answer.
Staging users get signed out after one page. Why?
Serve staging over HTTPS so the Secure session cookie persists.
You get the fix
Serve staging over HTTPS. The answer names the cause and the repair, but none of the checks that separated it from a bad guess.
The student learns the destination and has to reconstruct the investigation.
- +The root cause
- +The final repair
- −The evidence trail
- −Every competing next step
AccessAnyone with the product open.
Distillation through an api used to mean learning the destination. Now it means learning the route.
One lab trained a small reasoning model on traces from a competitor's model and published that those traces were about as useful as the leading open alternative's. A lab saying something flattering about a competitor's product is rare enough to take seriously as a measurement.
But a trace is one path, written down. The approaches the teacher considered and dropped leave nothing behind. It's far more than the destination was, and it still isn't the shape.
Who has the file
The two methods aren't better and worse. They're two amounts of access.
At least one large lab has described training a shipping model from an enormous internal teacher that was never released as a product. A model built only to teach other models. That's what you do when you own the file.
Everyone else gets text. Reasoning traces made that text far more valuable than it was two years ago, and it's still one sample per sentence.
Racing to the frontier
This is where the technique stops being a training detail and becomes an argument.
Pushing the frontier forward is enormously expensive. Copying what the frontier says is not. So every time somebody pays to move the line, everything downstream gets cheaper at the same moment. It also gets there second.
That's the version of the story you usually hear, and on its own it's too simple. Nobody who ships a serious model just fine-tunes on somebody else's answers and calls it done. Traces go in alongside a corpus that lab spent years assembling, into weights of its own shape and size, followed by post-training nobody else has run. The copied part is an ingredient. Sometimes a big one. Never the whole recipe.
building itself · click any ingredient
Distilled traces
Gets you: A large head start. Answers and reasoning traces from a stronger model are the cheapest quality data anyone has ever had access to.
Can’t: Reach past the model it came from. You are fitting to another model's outputs, so its ceiling is your ceiling, and its mistakes are in your training set.
That's also why "second" isn't permanent. The frontier isn't one ladder with everyone on different rungs of it.
One open lab's recent flagship shipped with a genuinely different architecture. A reworked attention scheme, memory pulled apart from computation. It also replaced the reinforcement-learning stage with distillation from a panel of the lab's own specialist teachers, and they published all of it. That's distillation used as an internal tool, by a lab that owns every file involved, wrapped in methods nobody had shipped before.
So the line moves sideways as often as it moves up. The lab that was behind on the old recipe isn't necessarily behind on the new one, and the model everyone was copying last year stops being the thing worth copying. A frontier defined by whoever spent the most is a much less stable thing than it sounds.
None of which makes the dispute go away. Anthropic's position is the clearest one published, and it draws the line in a specific place: distillation is described there as a widely used and legitimate training method, one that frontier labs routinely use on their own models to ship smaller and cheaper versions of them. The objection is not to the technique. It's to acquiring another lab's capabilities through fraudulent accounts and circumvented terms of service, at industrial scale, to skip the bill.
Through 2026 several labs have made public allegations along exactly those lines, in company posts and in letters to legislators, describing campaigns that ran tens of thousands of fake accounts and harvested tens of millions of exchanges. I'm not going to relitigate who did what here. Those are allegations, most of the accused have not answered them publicly, and none of it has been adjudicated. The mechanism is the interesting part, and it's the same one this whole post has been describing: somebody typed a great many questions into somebody else's model and trained on the answers.
The part that doesn't resolve
Both things are true at once. Distillation is how the field makes models small enough to be useful, including inside every lab, on their own models, with their own weights. It's also a way to shortcut toward capabilities somebody else paid to find. Almost any rule you could write to stop the second one lands on the first too, which is why the argument is about permission and scale rather than about the technique.
What that means if you just run models
Almost every model I run locally has traces from something I couldn't afford to run myself somewhere in its history. I picked them because they're fast and they fit.
That's a real trade and worth making with your eyes open. It isn't the flat "you're running a photocopy" that people assume. The good local models are the ones where the copied part was the floor rather than the plan, where somebody put their own data, their own architecture and their own post-training on top of it.
You can check that before you download 30 GB. Read what the lab published about how the model was made. Labs that did the work tend to write about it.
So the question isn't whether there's distillation in there. There is. It's what else is.