Pre-training via next-token prediction is 'magical' because it enables representation learning—models learn models of the world rather than mere statistical artifacts—which allows them to generalize and apply knowledge to novel problems, rather than just learning surface-level patterns.
factualpending
Speaker
Leopold AschenbrennerEvidence Quote
“Predicting the next token lets the model learn incredibly rich representations... Rather than just learning statistical artifacts, the models learn models of the world.”
Source
Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history— Dwarkesh PatelCreated: 8/12/2026, 6:39:27 PM
Your notes
Loading notes...