factual
Distillation from a big model boosts small models
It is easier to elicit powerful capabilities like reasoning with RL training on a bigger model, then distill it down to a smaller model; AI2 reproduced this with Tülu on Llama 405B, and the DeepSeek paper described the same pattern, showing large room for others to replicate these results cheaply.
factualpending
Speaker
Nathan LambertEvidence Quote
“this is something that the DeepSeek paper talked about as well is at this bigger model, it’s easier to elicit powerful capabilities with this RL training.”
Created: 6/13/2026, 3:31:25 AM
Your notes
Loading notes...