next-token prediction
The training objective of language models that predicts the most likely next word or token given preceding text.
As defined in AMA | April 2025
Other definitions
A self-supervised learning task where models are trained to predict the next word or token in a sequence given previous tokens, used to pre-train large language models
Yann LeCun's $1B Bet Against LLMs [Part 1]
Where it comes up · 10
- Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality3 claims
- Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence2 claims
- Yann LeCun's $1B Bet Against LLMs [Part 1]2 claims
- 230 | Raphaël Millière on How Artificial Intelligence Thinks2 claims
- 286 | Blaise Agüera y Arcas on the Emergence of Replication and Computation2 claims
- Andrej Karpathy — AGI is still a decade away1 claim
- Gaurav Misra & Dwight Churchill - Building Captions - [Invest Like the Best, EP.405]1 claim
- Joscha Bach1 claim
and 2 more
Most important claims · 10 of 16Search all claims for “next-token prediction” →
- Diffusion models generate video by starting from pure noise and, at each denoising step conditioned on text, progressively revealing structure such as a man in a blue shirt, unlike next-token prediction, which predicts the next word from prior context.Gaurav Misra & Dwight Churchill - Building Captions - [Invest Like the Best, EP.405]
- Next-token prediction is not mere imitation that caps at human ability; because predicting the next token well requires understanding the underlying reality that produced the text, a sufficiently smart base network can extrapolate how a hypothetical person of far greater insight, wisdom, and capability would behave, even if such a person doesn't exist, deducing it from the thoughts, feelings, and ideas implicit in the behavior of regular people.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Saying that language models 'only do next-word prediction' is misleading: to predict the next word well across virtually any linguistic context, a model may have to acquire sophisticated capacities, just as describing a human as 'only maximizing inclusive genetic fitness' would be a category mistake that fails to capture reasoning and thinking.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Self-supervised learning achieved success in language (next-token prediction) much faster than in vision and other modalities, suggesting language is a particularly rich and informative domain for self-supervised pre-training.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Scaling laws describe how the log of next-word prediction accuracy improves, but there is a separate, complicated challenge of linking next-word prediction accuracy to reasoning capability; the link exists but other methods, such as reasoning tokens, may yield more reasoning per unit of effort.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Today's AI models are genuinely intelligent, and rather than LLMs being mystically more than next-token prediction, brains are fundamentally about predicting the future; doing that prediction well requires generalization, internal representations, and sophisticated modeling, so 'just predicting next tokens' conceals great beauty and complexity.286 | Blaise Agüera y Arcas on the Emergence of Replication and Computation
- Task-specific AI built via supervised learning can at best do its defined task, which is why narrow systems like speech, face, and character recognition were not truly AI; real intelligence emerged only when models stopped optimizing for specific tasks and instead modeled freeform internet text, because language is a microcosm of everything in our umwelt worth talking about and next-token prediction builds a model of that entire distribution.286 | Blaise Agüera y Arcas on the Emergence of Replication and Computation
- The Transformer architecture, initially designed for language translation by Google, was repurposed by OpenAI for next-token prediction on self-supervised pre-training, becoming the foundation for GPT and all modern large language models.Yann LeCun's $1B Bet Against LLMs [Part 1]
- AI models trained with reinforcement learning rewarded for correct answers (rather than just next-token prediction) show large performance gains and exhibit emergent human-like behaviors such as spontaneous self-correction ('oops I made a mistake'); whether such emergent behaviors correlating with intelligence represent mere regurgitation or something genuinely more interesting is an open and fascinating question.AMA | April 2025
- Dealing with the real physical world is much more difficult than dealing with language, therefore the architectures needed for systems that can handle the real world must be completely different from current LLM architectures based on next-token prediction.Metas AI Boss Says He DONE With LLMS...