normative
Next-token training underuses each token
Current pre-training likely doesn't extract maximal value from every token, because predicting the next token from previous ones isn't really how people learn; the model should think much harder on some tokens than others (do a lot more work when reaching 'the answer is' than 'the'), learn data forwards and backwards every which way, and use techniques like dropout and many epochs—so the claim that we're running out of text data is doubtful since far more capability can be extracted from existing text.
normativepending
Speaker
Jeff DeanEvidence Quote
“we can get a lot more capable models out of the text data that does exist.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...