autoregressive
Generating output one token at a time, each conditioned on all previously generated tokens; the dominant paradigm for state-of-the-art LLMs.
As defined in Ai Sota #2026
Other definitions
A model generation approach where outputs are generated sequentially, with each output fed back as input to generate the next output (used in language models)
Yann LeCun's $1B Bet Against LLMs [Part 1]
Where it comes up · 8
- The Shape of AI to Come! Yann LeCun at AI Action Summit 20253 claims
- Illustrated Guide to Transformers Neural Network: A step by step explanation2 claims
- Yann LeCun Says LLMs Have 2 Years Left…1 claim
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]1 claim
- Yann LeCun's $1B Bet Against LLMs [Part 1]1 claim
- Ai Sota #20261 claim
- Deepseek Dylan Patel Nathan Lambert1 claim
- The Mastermind Behind GPT-4 and the Future of AI | Ilya Sutskever1 claim
Most important claims · 10 of 11Search all claims for “autoregressive” →
- Current LLMs operate by autoregressive token prediction where a causal transformer repeatedly takes a window of previous tokens and predicts the next token, and this creates a fundamental problem: each token prediction has some probability of being wrong, and if one is, there is no way to fix it afterwards, leading to exponential divergence and hallucination issues.The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- Text diffusion models generate all tokens in a completion in parallel by iterative de-noising (like BERT-style masked filling), promising much faster generation, but the trade-off is quality: papers show that to match autoregressive quality you must crank up de-noising steps until you spend the same compute, and tool use mid-generation is tricky—so diffusion will serve quick, cheap, at-scale tasks rather than replace autoregressive LLMs.Ai Sota #2026
- Inference in JEPA-based systems becomes a search process over predicted futures rather than autoregressive prediction, fundamentally changing how these systems generate behavior.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Current LLMs operate through autoregressive token prediction—producing one token at a time by predicting the next token given a fixed window of previous tokens—which creates exponential divergence because each token prediction has some probability of error, and once an incorrect token is generated, there is no way to correct it, leading to hallucinations.The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- The decoder's first multi-headed attention layer operates differently from the encoder because the decoder is autoregressive and generates sequences word-by-word, requiring it to prevent attending to future tokens that haven't been generated yet.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Output tokens are roughly four times more expensive than input tokens because input (prefill/prompt) can be processed in parallel as a batch, whereas output is autoregressive — each generated token requires re-reading the whole model and KV cache before generating the next, a non-parallel operation.Deepseek Dylan Patel Nathan Lambert
- Although transformers generate one token at a time, there is strong evidence they plan ahead internally, similar to humans who form internal plans before speaking, and pretraining helps develop this planning skill.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Current autoregressive transformers already handle predicting high-dimensional distributions with uncertainty effectively, as evidenced by their ability to predict the next page in a book or generate images, contradicting claims that new architectures are necessary for this capability.The Mastermind Behind GPT-4 and the Future of AI | Ilya Sutskever
- Intelligent behavior requires two essential characteristics: the ability to predict consequences of your actions, and the ability to plan through optimization and search to find sequences of actions that accomplish goals, neither of which LLMs can do since they operate through autoregressive token prediction rather than search.Yann LeCun Says LLMs Have 2 Years Left…
- The decoder is autoregressive, meaning it takes in the list of previous outputs as inputs along with encoder outputs that contain attention information from the input, and it stops decoding when an end token is generated.Illustrated Guide to Transformers Neural Network: A step by step explanation