autoregressive LLM
A large language model constrained to predict the next token using only the tokens before it, generating text one token at a time by sampling from a probability distribution over the vocabulary.
As defined in Research Pioneer
Where it comes up · 1
- Research Pioneer3 claims
Most important claims · 3 of 3Search all claims for “autoregressive LLM” →
- Autoregressive LLMs cannot achieve human-level intelligence because they lack or only primitively possess four essential characteristics of intelligent behavior: understanding the physical world, persistent memory, reasoning, and planning.Research Pioneer
- Building world models by training generative models to predict future video frames fails because you cannot represent probability distributions over high-dimensional continuous spaces (all possible frames) the way you can over a finite dictionary of text tokens; the world is too rich in unpredictable detail.Research Pioneer
- LLM hallucination arises because in autoregressive prediction each token has a non-zero probability of taking the output outside the set of correct answers, and assuming these errors are roughly independent, the probability of remaining correct decreases exponentially with the number of tokens produced.Research Pioneer