A language model is much better at learning about the world and building representations of ideas, concepts, people, and processes during pre-training than it is at producing good outputs, and there are technical reasons for this asymmetry that explain why hallucinations occur and why reinforcement learning from human feedback targets the output level.
causalpending
Speaker
Ilya SutskeverEvidence Quote
“It is indeed the case that these neural networks are they do have a tendency to hallucinate. But that's because a language model is great for learning about the world, but it is a little bit less great for producing good outputs.”
Created: 8/10/2026, 10:48:45 PM
Your notes
Loading notes...