Richard Sutton – Father of RL thinks LLMs are a dead end
What this covers
Rich Sutton, a foundational figure in reinforcement learning, sits down with Dwarkesh Patel to argue that large language models represent a dead end in AI development. Sutton contends that LLMs learn from mimicking human-generated text rather than from grounded experience with external goals and measurable feedback. His core claim is that genuine intelligence requires agents to interact with the world, receive reward signals tied to real outcomes, and continually refine their behavior through trial-and-error learning—a framework he traces through animal cognition, computational theory, and decades of AI research. The conversation moves between Sutton's technical definitions of intelligence and learning, concrete examples from game-playing AI, and broader implications for how superintelligence might emerge.
Sutton develops several interlocking arguments. He distinguishes reinforcement learning from the LLM approach by emphasizing that RL systems have goals and ground truth—a defined notion of correctness against which feedback arrives—while next token prediction lacks both. He frames Temporal Difference Learning as the mechanism by which agents learn to predict long-term value and adjust actions accordingly, comparing it to human intuition about progress. He also invokes The Bitter Lesson, the historical pattern showing that scalable, experience-driven methods eventually outcompete human-knowledge-laden approaches, predicting LLMs may be the next victim. The conversation touches on generalization failures in deep learning, Moravec's paradox (why AI excels at math yet lacks continual learning), and Big World Hypothesis (why agents must learn idiosyncratic details online rather than receive pre-loaded knowledge). Dwarkesh Patel also contributes a normative stance on instilling values in increasingly powerful AI systems rather than attempting detailed control.
Sutton argues that LLMs are a dead end because they mimic human-generated text rather than learning from grounded experience with goals and ground truth, and that genuine intelligence requires continual experiential reinforcement learning—a paradigm that will eventually supersede the LLM approach per the Bitter Lesson.
- LLMs lack goals, ground truth, and the ability to predict and be surprised by world outcomes, so they cannot truly learn continually
- Animals (including humans) learn primarily through trial-and-error and prediction, not supervised imitation, making experiential RL the right foundation
- Historically, scalable experience-and-computation methods have always eventually beaten human-knowledge-laden approaches (the Bitter Lesson)
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
In the experiential paradigm, sensation, action, and reward form an ongoing stream that is the foundation of intelligence; intelligence is altering actions to increase rewards in the stream, and knowledge is statements about the stream that can be tested against it and learned continually.
“It says that this is the foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase the rewards in the stream.”
Temporal difference learning lets an agent learn from short-term progress toward a distant goal: a value function predicts the long-term outcome, and when an intermediate action raises that prediction, the increase immediately reinforces the action that caused it—analogous to a human feeling more likely to succeed at a startup and rewarding the steps that produced that feeling.
“You do that by having a value function which predicts the long-term outcome. Then if you take the guy’s pieces, your prediction about the long-term outcome is changed... that increase in your belief immediately reinforces the move that led to taking the piece.”
Current AI is not achieving genuine transfer; good performance requires generalizing well from one state to another, but we have very few automated techniques to promote transfer and none are used in modern deep learning—when generalization is good it is because human researchers sculpted a representation, not because the algorithms produce it.
“We’re not seeing transfer anywhere. Critical to good performance is that you can generalize well from one state to another state. We don’t have any methods that are good at that.”
Reinforcement learning is basic AI focused on understanding your world, whereas large language models are about mimicking people and doing what people say you should do; they are not about figuring out what to do.
“Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They’re not about figuring out what to do.”
LLMs cannot support genuine prior knowledge or continual learning because there is no ground truth or goal in their setup: there is no definition of the 'right thing to say,' so they get no real feedback, whereas in reinforcement learning the right thing to do is defined as the action that gets reward.
“There’s no ground truth in large language models because you don’t have a prediction about what will happen next.”
If we understood a squirrel, we would be almost all the way to understanding human intelligence, because humans are animals first and the language part is just a small veneer on the surface; what we have in common with animals is more important than what distinguishes us.
“If we understood a squirrel, I think we’d be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.”
Having a goal is the essence of intelligence; something is intelligent if it can achieve goals, following John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. Without a goal there is no sense of better or worse, so a pure prediction system is not intelligent.
“I like John McCarthy’s definition that intelligence is the computational part of the ability to achieve goals.”
Rather than dictating exactly what AIs will do or planning the world a century out, the appropriate approach—analogous to raising children—is to instill robust, steerable, high-integrity values (honesty, refusing harmful goals) so that when AIs end up in positions of power they behave well, and to seek change that is voluntary rather than imposed.
“it’s quite important to give them robust and steerable and prosocial values.”
Succession to digital or augmented intelligence is inevitable, by a four-part argument: (1) there is no unified global authority or consensus on how the world should be run; (2) researchers will eventually figure out how intelligence works; (3) we will not stop at human-level but reach superintelligence; and (4) over time the most intelligent things will gain resources and power.
“I do think succession to digital intelligence or augmented humans is inevitable. I have a four-part argument.”
AlphaGo was in essence a scaling up of Gerry Tesauro's TD-Gammon, which used temporal difference learning to play backgammon and beat the world's best players; AlphaGo added an innovation in search but was not fundamentally surprising, while AlphaZero (which used TD) succeeded across games and impressively played chess by patiently sacrificing material for positional advantage.
“The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro did reinforcement learning, temporal difference learning methods, to play backgammon.”
It is striking that AI systems can solve difficult math problems (which almost no animal can do) yet lack continual learning (which all mammals have), illustrating Moravec's paradox where what is hard for AI versus humans is often inverted.
“It’s quite interesting that we have something that all mammals have, but our AI systems don’t have.”
The base common model of an agent has four parts: a policy (what to do in the current situation), a value function (a learned number indicating how well things are going, used to adjust the policy), a perception component (constructing the state representation), and a transition model of the world (predicting the consequences of actions), the last of which is learned richly from all sensation rather than from reward.
“We need a policy... We need a value function... Then there’s also the perception component... The fourth one is the transition model of the world.”
The big world hypothesis explains why humans become useful on the job: the world is so huge that an agent's particular idiosyncratic part of it cannot be anticipated or pre-loaded in advance, so the agent must learn it online during its life, contradicting the LLM dream of teaching the agent everything beforehand.
“the big world hypothesis seems very relevant. The reason why humans become useful on the job is because they are encountering their particular part of the world. It can’t have been anticipated and can’t all have been put in in advance.”
Supervised learning is not part of how animals learn: animals have no examples of desired behavior, only examples of things that happened in sequence and of actions with their consequences; basic animal learning consists of prediction and trial-and-error control, not imitation or supervised learning.
“supervised learning is not part of the way animals learn. We don’t have examples of desired behavior.”
Although you can always start with human knowledge before adding scalable methods, in practice this has always turned out badly: people get locked into the human-knowledge approach and eventually have their lunch eaten by methods that are truly scalable from experience and computation—this is the Bitter Lesson, and LLMs may be the next instance.
“People get locked into the human knowledge approach... They get their lunch eaten by the methods that are truly scalable.”
Beyond an arbitrary task reward (winning chess, getting nuts, avoiding pain and acquiring pleasure), a general learning agent should have a reward component tied to increasing its understanding of its environment—a form of intrinsic motivation.
“I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.”
Next-token prediction is not a real goal because it does not change the world: tokens come at the model and predicting them does not influence them, so the system is just predicting and being satisfied with its accuracy rather than pursuing an external-world objective.
“That’s not a goal. It doesn’t change the world. Tokens come at you, and if you predict them, you don’t influence them.”
Cultural knowledge (like complex multi-step skills such as hunting seals) that cannot be reasoned through must be transmitted across generations via imitation, but this imitation is a small thing layered on top of the basic trial-and-error and prediction learning that we share with animals.
“No, I think about it the same way. Still, it’s a small thing on top of basic trial-and-error learning, prediction learning.”
The long-standing AI controversy between general-purpose 'weak' methods (search and learning) and human-knowledge-laden 'strong' symbolic methods has been decisively won by the weak methods; learning and search have won the day, validated by surprises like LLM language ability, AlphaGo, and AlphaZero.
“I think the weak methods have just totally won. That’s the biggest question from the old days of AI, what would happen. Learning and search have just won the day.”
Excelling at math problems does not generalize to modeling the physical world because math is computational and more like standard planning where the goal of finding a proof can be given, whereas the empirical world's consequences must be learned through experience.
“Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned.”
When LLMs get problems right, it may be because there is only one way to solve them, so finding that single solution is not generalization; genuine generalization is when multiple solutions exist and the system reliably picks the one that generalizes well, which the algorithms do not ensure.
“If there’s only one answer and you find it, that’s not called generalization... But generalization is when it could be this way, it could be that way, and they do it the good way.”
LLMs do not have a genuine world model; they have the ability to predict what a person would say, not the ability to predict what will happen in the world, because they are mimicking things (people) that have world models.
“A world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don’t have the ability to predict what will happen.”
Deep learning generalizes poorly in that training on a new thing often catastrophically interferes with previously learned things; generalization always happens but can be good or bad, and we lack algorithms that reliably cause generalization to be good.
“if you train on some new thing, it will often catastrophically interfere with all the old things that you knew. This is exactly bad generalization.”
AlphaZero improved on AlphaGo precisely by not using human knowledge and instead learning purely from experience, which is why bringing in another agent's expertise to teach an agent is the wrong path when experience-only learning has worked so well.
“And the way AlphaZero was an improvement was that it did not use human knowledge but just went from experience.”
Infants do not learn primarily by imitation; they learn by trying things—waving hands, moving eyes, making sounds—for which there are no targets or examples, indicating their behavior is generated by trial-and-error rather than copying.
“When I see kids, I see kids just trying things and waving their hands around and moving their eyes around. There’s no imitation for how they move their eyes around or even the sounds they make.”
In a continual learning setup, the tacit context an agent picks up (about clients, company, preferences) goes into the network's weights as a learned policy specific to the environment, rather than into a context window as in LLMs; the agent learns from all sensory data, not just from reward.
“in a continual learning setup, it just goes into the weights.”
Unlike humans where each new child must relearn the world from scratch, a digital intelligence could learn once and be copied into the next instance as a starting place, allowing knowledge to be shared across instances—a huge savings that is more important than trying to learn from people.
“with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place. This would be a huge savings.”
In an age of digital spawning and knowledge re-incorporation, a major concern will be corruption: pulling external information directly into your central mind could carry viruses or hidden goals that warp, take over, or destroy you, making cybersecurity a central problem rather than knowledge transfer being trivially easy.
“you’ve just read a whole bunch of bits into your mind, and they could have viruses in them, they could have hidden goals, they can warp you and change you. This will become a big thing.”
A key question unique to digital intelligences is whether an AI given more compute should make itself more computationally capable or spawn a copy to learn something new elsewhere and report back; this raises the unresolved question of whether a changed offshoot can be reincorporated into the original or whether it has diverged too much.
“you’re an AI, you get some more computer power. Should you use it to make yourself more computationally capable? Or should you use it to spawn off a copy of yourself to go learn something interesting on the other side of the planet... and then report back to you?”
Large language models are a poor vehicle for science because they are so complex and have been fed so much data that we cannot know what prior information they had, making their successes uncontrolled and unexplained.
“This is one reason why they’re not a good way to do science. It’s just so uncontrolled, so unknown.”
One's attitude toward radical change (including AI succession) largely reflects how good one thinks the current situation is; Sutton considers the present human situation imperfect and pretty bad, so he is more open to change, noting that most of humanity has little influence over major events like control of nuclear weapons or nation states.
“If you think the current situation is really good, then you’re more likely to be suspicious of change... I think it’s imperfect. In fact, I think it’s pretty bad. So I’m open to change.”
Whether we regard future designed intelligences as our offspring to be celebrated or as alien entities to be horrified by is genuinely our choice, even though the matter feels too strongly held to be a choice—a contradiction Sutton finds interesting.
“It’s our choice whether we should say, “Oh, they are our offspring and we should be proud of them”... Or we could say, “Oh no, they’re not us and we should be horrified.””
Humans should avoid a sense of entitlement that the future must evolve the way they want, recognizing our limited control over the long-term future; a healthier counterbalance is to focus on the more controllable local goals of our own lives and families, since competing demands to steer the global future generate conflict.
“I think we want to avoid the feeling of entitlement... How much control should a particular species on a particular planet have over it?”
The rise of designed intelligence marks one of the four great stages of the universe (dust to stars, stars to planets, planets to life, and now replicators to designed entities): we are transitioning from a world of replicators we don't understand to designed AIs we do understand and can change at will, and future entities may be entirely designed rather than replicated.
“We’re entering the age of design because our AIs are designed... This is a key step in the world and in the universe. It’s the transition from the world in which most of the interesting things that are, are replicated.”
If we have already worked out and reached AGI, then we are done, because using AGI to get AGI is circular, and any knowledge an AGI imparts would not be superhuman if the AGI itself is not superhuman.
“Then we’re done. ... But you’re using it to get AGI again.”
Large language models have a goal, namely next-token prediction.
“No, they have a goal. ... Next token prediction.”