A language model is much better at learning about the world and building representations of ideas, concepts, people, and processes during pre-training than it is at producing good outputs, and there are technical reasons for this asymmetry that explain why hallucinations occur and why reinforcement learning from human feedback targets the output level.

causalpending

Speaker

Ilya Sutskever

Evidence Quote

“It is indeed the case that these neural networks are they do have a tendency to hallucinate. But that's because a language model is great for learning about the world, but it is a little bit less great for producing good outputs.”

Source

The Mastermind Behind GPT-4 and the Future of AI | Ilya Sutskever— Eye on AI
Created: 8/10/2026, 10:48:45 PM

Your notes

Loading notes...