factual

Distillation from a big model boosts small models

It is easier to elicit powerful capabilities like reasoning with RL training on a bigger model, then distill it down to a smaller model; AI2 reproduced this with Tülu on Llama 405B, and the DeepSeek paper described the same pattern, showing large room for others to replicate these results cheaply.

factualpending

Speaker

Nathan Lambert

Evidence Quote

“this is something that the DeepSeek paper talked about as well is at this bigger model, it’s easier to elicit powerful capabilities with this RL training.”

Source

Deepseek Dylan Patel Nathan Lambert— Lex Fridman Podcast
Created: 6/13/2026, 3:31:25 AM

Your notes

Loading notes...