factual
Pre-training shrinks as budget shifts to RL
Labs are being practical with fixed flops and cost budgets: pre-training is not where you want most of your flops, so models have gotten smaller in the pre-training stage but make it up in reinforcement learning and mid-training to get the most bang for the buck.
factualpending
Speaker
Andrej KarpathyEvidence Quote
“they make it up in reinforcement learning, mid-training, and all this stuff that follows”
Created: 6/13/2026, 3:26:44 AM
Your notes
Loading notes...