normative

Next-token training underuses each token

Current pre-training likely doesn't extract maximal value from every token, because predicting the next token from previous ones isn't really how people learn; the model should think much harder on some tokens than others (do a lot more work when reaching 'the answer is' than 'the'), learn data forwards and backwards every which way, and use techniques like dropout and many epochs—so the claim that we're running out of text data is doubtful since far more capability can be extracted from existing text.

normativepending

Speaker

Jeff Dean

Evidence Quote

“we can get a lot more capable models out of the text data that does exist.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...