causal

Pretraining via next-word prediction stores concepts

Pretraining a model to predict the next word across trillions of examples produces a model that is only moderately accurate (around 18 percent) but must store a large number of concepts internally, which is why it can predict next words in science papers or web text.

causalpending

Speaker

Gavin Uberti

Evidence Quote

“you have to have a huge number of concepts stored within that machine”

Source

Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]— Invest Like the Best
Created: 11/17/2026, 12:35:53 PM

Your notes

Loading notes...