Token Prediction
Machine learning approach where a model learns to predict the next token in a sequence from previous tokens, used as the training objective for language models
As defined in Metas AI Boss Says He DONE With LLMS...
Other definitions
The fundamental training objective of language models: predicting the next token (word or subword) in a sequence based on previous tokens, allowing models to learn language patterns and structure
How ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolutionThe task of predicting the next element (token) in a sequence given the previous context; the core objective for training language models
Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
Where it comes up · 4
- The Engineering Unlocks Behind DeepSeek | YC Decoded5 claims
- Metas AI Boss Says He DONE With LLMS...2 claims
- How ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolution1 claim
- Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History1 claim
Most important claims · 9 of 9Search all claims for “Token Prediction” →
- GPT models trained at scale with simple objectives (next-token prediction) can accomplish creative tasks like writing sonnets, generating poetry, and producing stylistically varied content that were long assumed to require human-level creativity and semantic understandingHow ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolution
- Cats perform complex planning and trajectory calculations (e.g., jumping on furniture) that involve abstract mental operations far more sophisticated than those achievable through language or token prediction.Metas AI Boss Says He DONE With LLMS...
- Multi-token prediction improves representation planning, allowing the model to pre-plan sequences for smoother and more coherent outputs during inference.The Engineering Unlocks Behind DeepSeek | YC Decoded
- The Transformer architecture was designed specifically for token prediction (next-word prediction in language models) by Google researchers in 2017; its remarkable capabilities emerged when coupled with massive data and compute, but this does not mean Transformers are the key ingredient for other AI tasks like logical reasoning or robotics.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Unlike traditional models that predict only the next token, Deep Seek V3 uses multi-token prediction (MTP) to anticipate multiple future tokens at each step, densifying training signals and providing more feedback per step for better data efficiency and faster learning.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Deep Seek V3 uses a mixture of experts architecture with 671 billion total parameters but only activates 37 billion parameters for each token prediction, compared to Llama 3's largest model which uses 405 billion parameters without mixture of experts, meaning V3 activates 11x fewer parameters per forward pass.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Multi-token prediction (MTP) modules can be repurposed for speculative decoding, reducing sequential processing steps and significantly speeding up generation during inference.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Transformers are not suitable architectures for predicting the physical world because they are fundamentally designed for token prediction rather than abstract representation learning.Metas AI Boss Says He DONE With LLMS...
- Deep Seek V3, combining multiple efficiency innovations (8-bit training, FP8 accumulation fix, mixture of experts, multi-head latent attention, multi-token prediction), is one of the most impressive base models on the market and has been available for some time (released December 2024).The Engineering Unlocks Behind DeepSeek | YC Decoded