causal
Pretraining via next-word prediction stores concepts
Pretraining a model to predict the next word across trillions of examples produces a model that is only moderately accurate (around 18 percent) but must store a large number of concepts internally, which is why it can predict next words in science papers or web text.
causalpending
Speaker
Gavin UbertiEvidence Quote
“you have to have a huge number of concepts stored within that machine”
Source
Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]— Invest Like the BestCreated: 11/17/2026, 12:35:53 PM
Your notes
Loading notes...