Distillation
A technique for separating components by heating and condensing; in Islamic food technology used primarily for essences rather than alcohol.
As defined in 147 | Rachel Laudan on Cuisine, Culture, and Empire
Other definitions
A technique for compressing the knowledge of a large, capable AI model into a smaller, faster model that can be deployed more cheaply while retaining much of the larger model's capability
Bret Taylor: Facebook, Salesforce, OpenAI, and Sierra
Where it comes up · 40
- Model Distillation: How to Steal a Billion-Dollar AI14 claims
- Adam Karr: Know Yourself, Pick Your Game, Build Your Blueprint (Lessons from a Janitor & Investing)9 claims
- Geoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 20245 claims
- The Shape of AI to Come! Yann LeCun at AI Action Summit 20255 claims
- Will AI outsmart human intelligence? - with 'Godfather of AI' Geoffrey Hinton4 claims
- Sam Altman - How to Make an Abundant Future - [Invest Like the Best, EP.484]4 claims
- Capital, Compute & AI Scaling3 claims
- Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch3 claims
and 32 more
Most important claims · 10 of 96Search all claims for “Distillation” →
- Efficiency has never stopped mattering in computing because someone always pays for it; while a model company can absorb inefficiency for a long time with a strong other business, in a vacuum more efficiency improves latency, cost, and scalability.Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]
- Islam and the religions of the Middle East exploded across Central Asia, West Africa, and East Africa between the 7th and 12th centuries—the next big event in culinary history—developing dramatic wheat cuisines (flat breads, noodles), keeping a form of sacrifice and meat-eating, avoiding alcohol, and advancing food technology with sugar and distillation for essences.147 | Rachel Laudan on Cuisine, Culture, and Empire
- Distillation is partly latent and unstoppable because AI-generated code and artifacts published on GitHub can serve as training signal, so preventing the diffusion of model capabilities is fundamentally impossible; only the speed is open.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Digital intelligences can share knowledge millions to billions of times more efficiently than humans: many identical copies process different data, each computes desired weight changes, and they average those changes (sharing weights or gradients) at trillions of bits per sharing—whereas humans transmit only about 100 bits per sentence via distillation (mimicking observed behavior), so 10,000 digital agents taking 10,000 different courses can all learn everything.Will AI outsmart human intelligence? - with 'Godfather of AI' Geoffrey Hinton
- Knowledge sharing between digital systems running different copies of the same model is extremely efficient because if you have two copies that have seen different data, they can share gradient updates by averaging weights across a trillion parameters, achieving vastly higher knowledge compression than distillationGeoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 2024
- Knowledge can be transferred between analog neural systems through distillation (having the system learn to match the outputs of a teacher system), which is how humans transfer knowledge—teachers speak and students adjust their weights to reproduce what the teacher said.Geoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 2024
- AI is inherently more difficult to protect than technologies like the atomic bomb because knowledge distillation allows lower-level AI models to extract insights from higher-level ones, the field employs far more people with weaker security protocols than the Pentagon or CIA, and AI models are embedded in consumer products and online making widespread access inevitable.AI’s New World Order: US-China, War, Job Loss | Tyler Cowen
- You can copy a frontier model without ever touching its weights by asking enough good questions through the API, saving the answers, and training a small model on those answers—a technique called distillation.Model Distillation: How to Steal a Billion-Dollar AI
- The proliferation of open-source models, especially Meta's LLaMA family in many sizes, lets small teams download, distill, fine-tune, and modify models to catch up to the frontier with clever techniques focused on a vertical or use case.Capital, Compute & AI Scaling
- The compute required to be first to achieve something is always much larger than the compute needed to catch up, because followers can use tricks like distillation and good initializations as crutches; AlphaGo Zero used ~3E23 FLOPS as a huge outlier, but Jang reproduced a strong Go bot for roughly $10K of rented compute.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch