distill
Distillation: compressing a large model's knowledge into a smaller, faster model so it can do inference quickly and cheaply.
As defined in AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
Where it comes up · 3
- AGI is still 30 years away — Ege Erdil & Tamay Besiroglu1 claim
- Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute1 claim
- What the Heck is Graph Engineering?1 claim
Most important claims · 3 of 3Search all claims for “distill” →
- Capabilities like reasoning look cheap and easy in retrospect (only ~1% of compute on RL) only because we stand on a huge stack of technology built over years; in 2015 building a reasoning model would have looked insurmountable, requiring internet-scale data, tens of thousands of GPUs, and distillation innovations — so agency may similarly look simple in hindsight while taking years of complementary innovation to actually achieve.AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
- Model size is not a guarantee of performance; ByteDance's model size gains would be most relevant if they follow through on their pledge not to distill from Western AI models.What the Heck is Graph Engineering?
- If AI takeoff is fast, the US wins because its labs are scaling compute to ~10 GW each, distillation from American models gets harder as work moves from visible reasoning chains to opaque automated white-collar output, and ROIC on its ~$1T data-center CapEx compounds; but if AI takes longer to reach high capability, China can catch up by building a fully vertical indigenized supply chain while the West's middling returns leave it exposed.Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute