factual

Pre-training shrinks as budget shifts to RL

Labs are being practical with fixed flops and cost budgets: pre-training is not where you want most of your flops, so models have gotten smaller in the pre-training stage but make it up in reinforcement learning and mid-training to get the most bang for the buck.

factualpending

Speaker

Andrej Karpathy

Evidence Quote

“they make it up in reinforcement learning, mid-training, and all this stuff that follows”

Source

Andrej Karpathy — AGI is still a decade away— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:44 AM

Your notes

Loading notes...