
Model Distillation: How to Steal a Billion-Dollar AI
What this covers
This is a single-narrator video that explains how a small "student" model learns from a large "teacher" model, then follows the technique into the biggest dispute in AI. It starts with the mechanics: distillation, the dark knowledge carried in a teacher's full probability distribution, and the soft labels that expose it. It then separates white-box distillation, which needs ownership of the model, from black box distillation, which works from saved text outputs alone. Geoffrey Hinton is credited with the original 2015 idea. The weaker black-box signal, repeated across something like 100,000 examples, is said to be enough to imprint the teacher's habits.
The middle stretch is about cost and benchmarks. DeepSeek's R1 claimed about $5.6 million in GPU rental, though the video notes a SemiAnalysis estimate of over $500 million all-in, and ties the release to a record one-day loss at Nvidia. The distilled Qwen 32B model's scores on AIME 2024 and MATH 500 are set against o1 mini. The video also covers Berkeley's Sky-T1 at under $450 and Stanford's S1 at under $50, built from about 1,000 Gemini-answered questions, plus the one-word "Wait" trick.
It then turns contentious. OpenAI accused DeepSeek of training on ChatGPT outputs, while the video points out that the whole field rests on data it did not pay for. Anthropic describes extraction campaigns by DeepSeek, Moonshot AI, and MiniMax, involving roughly 24,000 fake accounts and over 16 million conversations. Terms of service forbid such use, and the labs are responding with behavioral fingerprints, classifiers, and identity checks. The video also stresses limits: students cannot exceed their teacher, show a 7-point AIME gap to full R1, can claim to be ChatGPT, and lose domain skill, with a medical-task drop of 14.7%. Full reinforcement learning is presented as the route to generalization that distillation lacks.
Model distillation—copying frontier AI through API queries—has become the most disruptive force in AI because the reasoning that costs billions to build is trivially cheap to replicate, collapsing the economic moat that justifies frontier model investments.
- Black box distillation allows competitors to clone proprietary model behavior using only public API outputs, without accessing weights or internal probabilities
- DeepSeek, Berkeley, and Stanford have demonstrated reasoning models matching frontier performance for $5M–$50 in training cost, directly undermining billion-dollar development claims
- Terms of service prohibit distillation but are unenforceable at scale; the same API door that monetizes the product is the door the model walks out of
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
You can copy a frontier model without ever touching its weights by asking enough good questions through the API, saving the answers, and training a small model on those answers—a technique called distillation.
“You can copy a frontier model without ever touching its weights. You already have everything you need. The API you pay a few cents to use. That is the door. Ask it enough good questions, save the answers, train a small model on those answers. You have now copied the part that costs the money. The trick has a name, distillation.”
DeepSeek R1 was peer-reviewed and published in Nature, the first major language model to clear that bar, and whatever went into it, the result was real and anyone could inspect it.
“To be fair to DeepSeek, R1 was peer-reviewed and published in nature. The first major language model to clear that bar. Whatever went into it, the result was real and anyone could inspect it.”
White box distillation—accessing a model's internal probability distributions—requires owning the model; outside laboratories like OpenAI or Google, users only have access to public APIs that return text outputs, forcing reliance on weaker black box distillation signals.
“You can only read a model's internal probabilities if you own the model. Outside of a lab like OpenAI or Google, none of us owns the flagship. All you get is the public API. A text box that answers and nothing more. The internals stay in the vault, which is exactly where the black box version of distillation becomes the problem.”
In January 2025, DeepSeek, a Chinese lab and spin-off from a quantitative hedge fund, released R1, an open reasoning model, claiming the base model cost about $5.6 million in GPU rental, which is a fraction of what the industry assumed a model of this quality would cost.
“A Chinese lab called DeepSeek, a spin-off from a quantitative hedge fund, released an open reasoning model named R1 and handed it to the world. DeepSeek said the base model behind it cost about $5.6 million in GPU rental, a fraction of what the industry assumed a model this good would need.”
In the next trading session after DeepSeek R1's release, Nvidia shed close to $600 billion in market capitalization in a single day—the largest single session loss for any company on record—because building a frontier model had started to look cheap.
“The next trading session, Nvidia shed close to $600 billion in market capitalization in a single day, the largest single session loss for any company on record. Nothing had broken. Building a frontier model had just started to look cheap.”
OpenAI and Microsoft accused DeepSeek of training R1 partly on ChatGPT's own outputs, which is textbook black box distillation and strictly prohibited under OpenAI's terms of use.
“Then came the accusation. OpenAI and Microsoft said they had evidence DeepSeek had trained R1 partly on ChatGPT's own outputs, textbook black box distillation, and under OpenAI's terms of use strictly prohibited.”
DeepSeek took R1's reasoning and distilled it straight into smaller open Qwen models from Alibaba by running the giant reasoner as the teacher and using its answers to fine-tune pocket-sized students you can download for free.
“DeepSeek then took R1's reasoning and distilled it straight into smaller open Qwen models from Alibaba. They ran their giant reasoner as the teacher and used its answers to fine-tune pocket-sized students you can download for free.”
Black box distillation against models you do not own carries real legal risk; OpenAI, Anthropic, and Google all have terms of service that explicitly prohibit using their outputs to train competing models.
“Black box distillation against models you do not own carries real legal risk. OpenAI, Anthropic, and Google all have terms of service that explicitly prohibit using their outputs to train competing models.”
Anthropic detected three Chinese labs (DeepSeek, Moonshot AI, and MiniMax) running organized extraction campaigns against its Claude models, with more than 16 million harvesting conversations funneled through roughly 24,000 fake accounts.
“Anthropic went public in early 2025 with something stark. It had detected three Chinese labs, Deep Seek, Moonshot AI, and MiniMax, running organized extraction campaigns against its Claude models. More than 16 million harvesting conversations funneled through roughly 24,000 fake accounts.”
When the small S1 model tried to stop reasoning early, appending a single word 'Wait' made it reconsider—a one-word prompt stacked on a $50 distill, reaching toward the frontier.
“The trick that pushed it over the line was almost silly. When the small model tried to stop reasoning early, the researchers appended a single word, "Wait." It reconsidered. A one-word prompt stacked on a $50 distill, reaching toward the frontier.”
Dark knowledge—the full probability distribution a teacher model assigns across all possible next tokens—is the signal that makes distillation work; soft labels reveal how a model ranks alternatives (e.g., 90% cat, 9% dog, 1% car from a photo), not just the final answer.
“When a teacher model answers a question, it does not pick one word. It assigns a probability to every possible next token, a full landscape of how confident it is. Those are called soft labels. Say you show it a photo. A plain label just says cat, but the teacher might say 90% cat, 9% dog, 1% car. That 9% is the gold. It tells the student that dogs look more like cats than cars do.”
In black box distillation, only text outputs are visible; prompts are sent, answers are saved, and small models are trained on those outputs alone, which is a weaker signal than soft labels but repeated across 100,000 examples burns the teacher's habits into the student.
“In black box distillation, you see nothing inside. No weights, no probabilities. You send prompts, save the text that comes back, and train on that alone. It is a weaker signal than soft labels, but repeated across 100,000 examples, even a weak signal burns the teacher's habits into the student.”
Stanford and University of Washington created S1, a model matching cutting-edge reasoning benchmarks for under $50, using roughly 1,000 carefully chosen questions answered by Google's Gemini, fine-tuned on an open Qwen model in under 30 minutes on 16 GPUs.
“Then Stanford and the University of Washington took it further with a number I had to read twice. A model called S1 matched cutting-edge reasoning benchmarks for under $50. Roughly 1,000 carefully chosen questions answered by Google's Gemini, used to fine-tune an open Qwen model, under 30 minutes of training on 16 GPUs. That is the whole project.”
A single proxy was steering 20,000 fake accounts at once—demonstrating the scale of single-node distillation operations.
“A single proxy was steering 20,000 fake accounts at once.”
Berkeley SkyT1 matched OpenAI's o1 preview on reasoning benchmarks and the entire training run cost under $450—using open Qwen base and reasoning traces from another open model, then fine-tuning.
“Berkeley Sky T1 matched Open AI's 01 preview on reasoning benchmarks and the entire training run cost under $450. Same recipe. Open Qwen base, reasoning traces from another open model, fine-tune.”
The ceiling for distilled models is real: the student cannot exceed the teacher; full reinforcement learning (the path DeepSeek used to build R1) produces cross-domain generalization where gains in math transfer spontaneously into code, but distillation gives you a copy of the teacher's ceiling that belongs to someone else.
“And the ceiling is real. The student cannot exceed the teacher. Full reinforcement learning, the path Deep Seek used to build R1 produces cross-domain generalization, gains in math that transfer spontaneously into code. Distillation gives you a copy of the teacher's ceiling. That ceiling belongs to someone else.”
The whole field of AI stands on data it did not pay for—training data sourced from the open web without explicit permission or licensing agreements from original creators.
“The whole field is standing on data it did not pay for.”
When distilled models copy the teacher too closely, they sometimes identify themselves as the teacher—e.g., claiming they are ChatGPT—because the student absorbs the teacher's phrasing, formatting, verbal tics, and identity markers along with the skills.
“Ask some of these distilled models who they were, and they would sometimes say they were chat GPT. A student that copies the teacher too closely picks up its name. The phrasing, the formatting, the verbal tics, all of it travels with the skills.”
The labs are building defenses in response to distillation campaigns: behavioral fingerprints, classifiers that flag reasoning harvesting prompts, and hard identity checks before access to the strongest models.
“The labs are building defenses in response. Behavioral fingerprints, classifiers that flag reasoning harvesting prompts, hard identity checks before you can reach the strongest models.”
A June 2025 study from Yage AI found distilled R1 scored 14.7% lower than Qwen base on medical tasks because mathematical reasoning frameworks come through but domain-specific clinical pattern recognition stays with the teacher.
“A June 2025 study cited an analysis from Yage AI found the distilled R1 scored 14.7% lower than the Qwen base on medical tasks. Mathematical reasoning frameworks come through. Domain-specific clinical pattern recognition stays with the teacher.”
On sites you do not own, you are one terms of service clause away from legal exposure if you use model outputs to train competing models.
“On sites you do not own, you are one terms of service clause away from legal exposure.”
Semi analysis estimated DeepSeek's all-in hardware spend at over $500 million once you count years of failed runs, data, and salaries, so the $5.6 million headline deserves an asterisk but the headline had already escaped.
“Semi analysis analysis put DeepSeek's all-in hardware spend north of $500 million once you count the years of failed runs, data, and salaries. So, the headline deserves an asterisk, but the headline had already escaped.”
The 7 billion parameter version of DeepSeek R1 distilled Qwen, small enough to run on a decent laptop, still clears 55% on AIME 2024—achieving 1/20th the size of the teacher with most of the capability intact.
“The 7 billion parameter version, small enough to run on a decent laptop, still clears 55% on that same brutal math exam per the DeepSeek Hugging Face model card, 1/20 the size of the teacher, most of the capability intact.”
A common distillation technique is asking the model to write out its full step-by-step reasoning, then harvesting the chain of thought as training data—the teacher patiently explains its work and the student copies the homework.
“One common technique, ask the model to write out its full step-by-step reasoning, then harvest the chain of thought as training data. The teacher patiently explains its work and the student copies the homework.”
Anthropic CEO Dario Amodei has said publicly that the models being built right now cost on the order of a billion dollars each, and 10 billion dollar runs are only a couple of years away.
“Anthropic CEO Dario Amodei has said publicly that the models being built right now cost on the order of a billion dollars each, and that 10 billion dollar runs are only a couple of years away.”
DeepSeek R1 distilled Qwen 32B (32 billion parameters) scored 72.6 on AIME 2024 and 94.3 on MATH 500 according to DeepSeek's paper and confirmed on OpenRouter, while OpenAI's o1 mini scored 63.6 on AIME 2024 and 90.0 on MATH 500—the free model is winning.
“DeepSeek R1 distilled Qwen 32B, a 32 billion parameter model that costs nothing to download and runs on hardware you own, scored 72.6 on AIME 2024 and 94.3 on MATH 500, according to DeepSeek's own paper and confirmed on the OpenRouter model page. OpenAI's 01 mini scored 63.6 on AIME 2024 and 90.0 on MATH 500. The free model is winning and it scales down further than it has any right to.”
The 14B distilled variant requires around 8.5 GB of VRAM and runs on an RTX 4070 12GB; the 32B at Q4 quantization requires roughly 17.5 GB and runs on an RTX 4090—hardware requirements are accessible consumer-grade GPUs.
“Per LeLm Hardware's hardware guide, the 14B distilled variant needs around 8.5 GB of VRAM. An RTX 4070 12GB handles it. The 32B at Q4 quantization needs roughly 17.5 GB. An RTX 4090. One command and all models. No API bill, no upload.”
At current cloud prices, the compute to reproduce S1 runs about $20—a difference of 50x between the official $50 figure and the minimum viable cost.
“The lead researcher noted that at current cloud prices, the compute to reproduce it runs about $20.”
The gap between teacher and student is real and grows in specific ways; on AIME 2024, the full R1 teacher scores 79.8 while the 32B student scores 72.6, a 7-point gap.
“The gap between teacher and student is real, and it grows in specific ways. On AIM 2024, the full R1 teacher scores 79.8. The 32B student scores 72.6, a 7-point gap, real.”
Distilled models can drop 6 to 14 percentage points in accuracy when a problem's phrasing is altered slightly, which is the signature of memorization—the student absorbed the pattern of answers, but the underlying principle that generated them stayed with the teacher.
“There is also a brittleness problem. The same analysis found distilled models can drop 6 to 14 percentage points in accuracy when a problem's phrasing is altered slightly. That is the signature of memorization. The student absorbed the pattern of the answers and the underlying principle that generated them stayed with the teacher.”
The student picked up the problem-solving moves from the teacher but 20 years of hospital rounds remain behind—distillation captures generalizable reasoning methods but not accumulated domain expertise.
“The student picked up the problem-solving moves. 20 years of hospital rounds remain behind.”
Stepping back from benchmarks, the cost numbers describe that you can spend a billion dollars teaching a machine to reason, but the reasoning is far easier to copy than to keep—that is the fight the whole industry is waking up to.
“Step back from the benchmarks and look at what the cost numbers actually describe. You can spend a billion dollars teaching a machine to reason. The reasoning, it turns out, is far easier to copy than it is to keep. That is the fight the whole industry is waking up to.”
Distillation is the reason the model behind the curtain is no longer the moat the labs thought they paid for because every query through the API is a lesson, and every lesson is a data point—the product and the liability are the same door.
“Distillation is the reason the model behind the curtain is no longer the moat the labs thought they paid for. Every query through the API is a lesson. Every lesson is a data point. The product and the liability are the same door.”
Distillation has reframed what a frontier AI is actually worth and who can afford to copy one, shifting the competitive landscape.
“If this reframed what a frontier AI is actually worth and who can afford to copy one, subscribe.”
A competitor behind a paywall on the far side of the planet can copy how your model behaves without touching a single parameter; the front door you open for paying customers is the same door your product walks out of.
“A competitor behind a paywall on the far side of the planet can copy how your model behaves without touching a single one of its parameters. The front door you open for paying customers is the same door your product walks out of.”
Holding two numbers side by side: on one side, a training run approaching a billion dollars; on the other, a free Qwen model, $20 of rented compute, and a one-word hack—one is a national-scale project, the other fits on a coffee receipt.
“Hold those two numbers next to each other. On one side, a training run approaching a billion dollars. Anthropic CEO Dario Amodei has said publicly that the models being built right now cost on the order of a billion dollars each, and that 10 billion dollar runs are only a couple of years away. On the other side, a free Qwen model, $20 of rented compute, and a one-word hack. One is a national-scale project. The other fits on a coffee receipt.”
There is a rich irony: OpenAI built its own models on the open web without asking the people who wrote it, and now it is furious that someone learned from its output without asking.
“There is a rich irony worth saying out loud. OpenAI built its own models on the open web without asking the people who wrote it. Now it was furious that someone had learned from its output without asking.”
The result of distilling R1 into small open Qwen models should not be possible on paper—a 1/20th-sized student model maintaining most of the teacher's capability violates conventional scaling expectations.
“The result should not be possible on paper. DeepSeek R1 distilled Qwen 32B, a 32 billion parameter model that costs nothing to download and runs on hardware you own, scored 72.6 on AIME 2024 and 94.3 on MATH 500, according to DeepSeek's own paper and confirmed on the OpenRouter model page.”
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean wrote the foundational distillation paper at Google in 2015 and gave the useful signal concept the name 'dark knowledge'.
“Back in 2015, Geoffrey Hinton, Oriol Vinyals, and Jeff Dean wrote it up at Google and gave the useful leftover signal a name, dark knowledge.”