factual
Scaling Unlocks Emergent Abilities Nonlinearly
Since GPT-3 (2020), simply scaling models—more parameters, layers, and data—has been sufficient to produce breakthroughs, following scaling power laws for next-word prediction; in terms of actual capacities, performance improves nonlinearly, with abilities like solving math problems or commonsense reasoning emerging in transition-like phases past certain sizes, with notable connections to the physics notion of emergence.
factualpending
Speaker
Raphael MilliereEvidence Quote
“there's this paper that was published by OpenAI in 2020 that finds this scaling power laws that look at how scaling the size of the model and the amount of data... leads to these improvements”
Created: 6/13/2026, 12:19:51 AM
Your notes
Loading notes...