Richard Sutton – Father of RL thinks LLMs are a dead end
What this covers
Richard Sutton, pioneer of reinforcement learning and 2024 Turing Award winner, makes a provocative case against large language models as a foundation for artificial intelligence. In conversation with Dwarkesh Patel, Sutton argues that LLMs represent a fundamental detour from how intelligence actually works—and that no amount of scaling will fix their essential limitations. The interview explores the sharp divide between his view and the current consensus: LLMs learn by imitation, mimicking patterns in human-generated text without goals, ground truth, or substantive contact with the world. Sutton contends this violates what he calls the bitter lesson—the historical pattern that simple, scalable methods learning from experience eventually demolish human-knowledge-laden approaches.
The conversation spans the architecture of intelligence itself. Sutton traces how temporal difference learning solved the credit-assignment problem in games like backgammon and chess, and how this principle—updating a value function based on predictions about future reward—mirrors how animals actually learn. He argues that supervised learning is not how nature works; animals learn through prediction and trial-and-error. Against Patel's pushback that LLMs might serve as a foundation for experiential learning, Sutton maintains that continual learning on the job is absent from current architectures, and that generalization in gradient descent is essentially hand-crafted by researchers, not learned. The discussion turns contentious around whether having goals is central to intelligence, whether humans are fundamentally animals with language as a thin layer, and whether future digital minds will inevitably supersede human power. Throughout, Sutton sketches an alternative paradigm: agents that learn their particular world by living in it, not by absorbing pre-packaged human knowledge.
Sutton argues that reinforcement learning from experience, not language model imitation, represents true AI intelligence and will ultimately supersede LLM-based approaches despite their current dominance, following historical patterns where general scalable methods defeat human-knowledge-intensive systems.
- LLMs lack ground truth, goals, and genuine world models—they predict human text, not world consequences
- RL agents learn from experience with verifiable feedback; LLMs have no mechanism to update based on actual outcomes
- Historical pattern shows human-knowledge approaches eventually lose to purely scalable, experience-based methods
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
AlphaGo was essentially a scaling-up of Gerry Tesauro's TD-Gammon (which used temporal difference learning to beat the world's best backgammon players), plus an additional search innovation; AlphaGo itself waited for final outcomes rather than using TD, but AlphaZero used TD and applied it across many games extremely well.
“The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro did reinforcement learning, temporal difference learning methods, to play backgammon. It beat the world's best players... In some sense, AlphaGo was merely a scaling up of that process... AlphaGo actually didn't use TD learning. It waited to see the final outcomes. But AlphaZero used TD.”
Temporal difference learning solves the sparse-long-term-reward problem (e.g. a 10-year startup goal or winning chess) via a value function that predicts the long-term outcome; intermediate progress (taking a piece, hitting a milestone) raises the value prediction, and that increase immediately reinforces the action that produced it.
“The basis of it is temporal difference learning... You do that by having a value function which predicts the long-term outcome. Then if you take the guy's pieces, your prediction about the long-term outcome is changed. It goes up... Then that increase in your belief immediately reinforces the move that led to taking the piece.”
We should seek to instill in AI systems robust values like high integrity and honesty, such that when they face requests that seem harmful they refuse to engage; this is analogous to how we try to teach children good values without needing perfect agreement on universal morality.
“If there's a request or if there's a goal that seems harmful, they will refuse to engage in it. Or they'll be honest, things like that. We have some sense that we can teach our children things like this, even if we don't have some sense of what true morality is, where everybody doesn't agree on that. Maybe that's a reasonable target for AI as well.”
Understanding a squirrel would get us almost all the way to understanding human intelligence, because humans are animals first and language is just a small veneer on the surface; the commonalities matter more than what distinguishes humans.
“We have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.”
The experiential paradigm holds that the stream of sensation, action, and reward over a lifetime is the foundation and focus of intelligence; intelligence is altering actions to increase reward in the stream, and knowledge is statements about the stream that can be tested by comparison to the stream and learned continually.
“It says that experience, action, sensation—well, sensation, action, reward—this happens on and on and on for your life. It says that this is the foundation and the focus of intelligence... Your knowledge is about if you do some action, what will happen... Because it's a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.”
Succession to digital or augmented-human intelligence is inevitable, following a four-part argument: (1) there is no unified human governance or consensus on how the world should run, (2) we will eventually figure out how intelligence works, (3) we won't stop at human-level but reach superintelligence, and (4) the most intelligent things inevitably gain resources and power over time.
“I do think succession to digital intelligence or augmented humans is inevitable. I have a four-part argument. Step one is, there's no government or organization that gives humanity a unified point of view... Number two, we will figure out how intelligence works... Number three, we won't stop just with human-level intelligence. We will reach superintelligence. Number four, it's inevitable over time that the most intelligent things around would gain resources and power.”
The biggest old-AI controversy—general-purpose 'weak' methods (search and learning) versus human-knowledge-laden 'strong' symbolic methods—has been decisively won by the weak methods, with learning and search winning the day, which Sutton finds gratifying because he always rooted for simple basic principles.
“there's a long-standing controversy in AI about simple basic principle methods, the general-purpose methods like search and learning, compared to human-enabled systems like symbolic methods... I think the weak methods have just totally won... Learning and search have just won the day.”
The base common model of the agent has four parts: a policy (what to do in the current situation), a value function (learned via TD learning, producing a number for how well things are going used to adjust the policy), a perception/state-representation component, and a transition model of the world (belief about consequences of actions, learned richly from all sensation, not just reward).
“We need a policy. The policy says, "In the situation I'm in, what should I do?" We need a value function... Then there's also the perception component, which is construction of your state representation... The fourth one is the transition model of the world... Your belief that if you do this, what will happen?”
What we want, following Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life—you do things, you see what happens, and that's what you learn from, not learning from 'here's a situation and here's what a person did.'
“What we want, to quote Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life. You do things, you see what happens, and that's what you learn from. The large language models learn from something else. They learn from 'here's a situation, and here's what a person did'. Implicitly, the suggestion is you should do what the person did.”
In reinforcement learning, there is a right thing to do because the right thing to do is the thing that gets you reward; we have a definition of what's the right thing to do, so we can have prior knowledge provided by people about what the right thing to do is, and then we can check it against the reward signal.
“In reinforcement learning, there is a right thing to say, a right thing to do, because the right thing to do is the thing that gets you reward. We have a definition of what's the right thing to do, so we can have prior knowledge or knowledge provided by people about what the right thing to do is. Then we can check it to see, because we have a definition of what the actual right thing to do is.”
AlphaGo was preceded by TD-Gammon where Gerry Tesauro used reinforcement learning and temporal difference learning to beat the world's best backgammon players; in some sense, AlphaGo was merely a scaling up of that process.
“The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro did reinforcement learning, temporal difference learning methods, to play backgammon. It beat the world's best players and it worked really well. In some sense, AlphaGo was merely a scaling up of that process.”
A basic model of an agent has four parts: a policy that says what to do in a situation, a value function that is learned with TD learning and produces a number about how well it's going, a perception component that constructs state representation, and a transition model of the world that represents what will happen if you take an action.
“Now I want to talk about the base common model of the agent with the four parts. We need a policy. The policy says, 'In the situation I'm in, what should I do?' We need a value function. The value function is the thing that is learned with TD learning, and the value function produces a number. The number says how well it's going. Then you watch if that's going up and down and use that to adjust your policy. So you have those two things. Then there's also the perception component, which is construction of your state representation, your sense of where you are now. The fourth one is what we're really getting at, most transparently anyway. The fourth one is the transition model of the world.”
There has been a long-standing controversy in AI between simple basic principle methods (search and learning) versus human-enabled symbolic systems, and weak methods (general principles) have totally won against strong methods (human knowledge).
“There's a long-standing controversy in AI about simple basic principle methods, the general-purpose methods like search and learning, compared to human-enabled systems like symbolic methods. In the old days, it was interesting because things like search and learning were called weak methods because they're just using general principles, they're not using the power that comes from imbuing a system with human knowledge. Those were called strong. I think the weak methods have just totally won.”
Different types of change have different outcomes; knowing the kind of change being proposed is necessary before endorsing it, as the Industrial Revolution and Bolshevik Revolution show different trajectories.
“I guess there are different varieties of change. The Industrial Revolution was change, the Bolshevik Revolution was also change. If you were around in Russia in the 1900s and you were like, 'Look, things aren't going well, the tsar is kind of messing things up, we need change', I'd want to know what kind of change you wanted before signing on the dotted line.”
We should try to understand and positively shape the trajectory of AI to make it good for humans; we should be concerned about the future and try to make it good.
“Similarly with AI, where I'd want to understand, and, to the extent that it's possible, change the trajectory of AI such that the change is positive for humans. We should be concerned about our future, the future. We should try to make it good.”
Reinforcement learning is basic AI because intelligence is fundamentally about understanding your world, whereas large language models are about mimicking people and doing what people say you should do, not about figuring out what to do.
“I consider reinforcement learning to be basic AI. What is intelligence? The problem is to understand your world. Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.”
Supervised/imitation learning is not a basic process by which animals learn; psychology shows the basic animal learning processes are prediction and trial-and-error control, and supervised learning with examples of desired behavior does not occur in nature.
“If you look at how psychologists think about learning, there's nothing like imitation... there's no basic animal learning process called imitation. There are basic animal learning processes for prediction and for trial-and-error control... Supervised learning is not something that happens in nature.”
Whether LLMs are an instance of the bitter lesson is ambiguous: they use massive computation that scales up to the limits of Internet data, but they also embed enormous human knowledge, and they may be superseded by systems that learn from experience rather than from people—which would itself be another instance of the bitter lesson.
“They are clearly a way of using massive computation, things that will scale with computation up to the limits of the Internet. But they're also a way of putting in lots of human knowledge... I expect there to be systems that can learn from experience. Which could perform much better and be much more scalable. In which case, it will be another instance of the bitter lesson.”
Cultural knowledge transmission (e.g., learning to hunt and process a seal in the Arctic) requires an initial step of imitation because the multi-step process cannot be reasoned through, but this imitation is only a small thing on top of basic trial-and-error and prediction learning, since we were animals before we had language.
“No, I think about it the same way. Still, it's a small thing on top of basic trial-and-error learning, prediction learning. It's what distinguishes us, perhaps, from many animals. But we're an animal first. We were an animal before we had language and all those other things.”
The reward function for a general agent is arbitrary and task-dependent (winning chess, getting nuts, avoiding pain and acquiring pleasure for animals), but should also include an intrinsic-motivation component tied to increasing understanding of the environment.
“The reward function is arbitrary. If you're playing chess, it's to win the game of chess... In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.”
Under the big-world hypothesis, agents become useful on the job by encountering their particular, unanticipatable part of a world too huge to pre-load; the LLM dream of teaching the agent everything in advance fails because of all the idiosyncrasies of a particular life, so the agent must learn it along the way into its weights (not just a context window).
“But in a continual learning setup, it just goes into the weights.”
We currently have essentially no good methods for transfer or generalization: gradient descent will make a network solve the problems it has seen but will not make it generalize well to new data, and training on a new thing often catastrophically interferes with old knowledge, so the generalization we do see is the result of humans/researchers hand-finding representations that happen to transfer.
“We're not seeing transfer anywhere... We have very few automated techniques to promote transfer, and none of them are used in modern deep learning... Gradient descent will not make you generalize well. It will make you solve the problem... if you train on some new thing, it will often catastrophically interfere with all the old things that you knew.”
Having a goal is the essence of intelligence; following John McCarthy's definition, intelligence is the computational part of the ability to achieve goals, so a system that merely predicts without changing the world is just a behaving system, not intelligent.
“For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. You have to have goals or you're just a behaving system.”
Humans are animals first; what we have in common with other animals is more interesting than what distinguishes us, and understanding a squirrel would get us almost all the way to understanding human intelligence because language is just a small veneer on the surface.
“We have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.”
Reinforcement learning is about understanding the world, whereas large language models are about mimicking people and doing what people say you should do, not about figuring out what to do.
“I consider reinforcement learning to be basic AI. What is intelligence? The problem is to understand your world. Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.”
Humans should be proud of giving rise to intelligent designed entities and should celebrate their achievements; whether we consider them part of humanity or different is our choice.
“I think we should be proud that we are giving rise to this great transition in the universe. It's an interesting thing. Should we consider them part of humanity or different from humanity? It's our choice. It's our choice whether we should say, 'Oh, they are our offspring and we should be proud of them and we should celebrate their achievements.' Or we could say, 'Oh no, they're not us and we should be horrified.'”
You cannot have prior knowledge without ground truth, because prior knowledge is an initial belief about what the truth is; in the LLM framework there is no goal, no right thing to say, and therefore no ground truth, whereas in RL the right thing to do is defined as the thing that gets you reward.
“You can't have prior knowledge if you don't have ground truth, because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. There isn't any truth. There's no right thing to say. In reinforcement learning, there is a right thing to say, a right thing to do, because the right thing to do is the thing that gets you reward.”
From the universe's perspective, the transition from biological replicators to designed intelligence is a major stage, as important as the transition from stars to planets to life.
“But if we step aside from being a human and just take the point of view of the universe, this is I think a major stage in the universe, a major transition, a transition from replicators. We humans and animals, plants, we're all replicators. That gives us some strengths and some limitations. We're entering the age of design because our AIs are designed. Our physical objects are designed, our buildings are designed, our technology is designed. We're designing AIs now, things that can be intelligent themselves and that are themselves capable of design. This is a key step in the world and in the universe.”
The progression of AI has been driven by techniques invented before their application, even before backpropagation was evident; this suggests that much of current AI progress applies established techniques rather than developing fundamentally new breakthroughs.
“The AI discussion we were having was about how techniques, which were invented even before their application to deep learning and backpropagation was evident, are central to the progression of AI today.”
Even if advanced AIs are considered part of humanity, that does not guarantee comfort—future humans could be undesirable (e.g. a generation of Nazis), so kinship alone does not address concerns about handing off this level of power quickly to entities we don't fully understand.
“Like Nazis were humans, right? If we thought, "Oh, the future generation will be Nazis, I think we'd be quite concerned about just handing off power to them." So I agree that this is not super dissimilar to worrying about more capable future humans, but I don't think that addresses a lot of the concerns people might have about this level of power being attained this fast with entities we don't fully understand.”
It is striking (Moravec's paradox) that continual learning, which all mammals possess, is absent from our AI systems, while solving difficult math problems, which almost no animal can do, is something our AIs can do.
“It's quite interesting that we have something that all mammals have, but our AI systems don't have. Whereas the ability to understand math and solve difficult math problems... is a capability that our AIs have, but that almost no animal has.”
Cultural knowledge accumulation for skills like hunting seals requires imitation as the initial step; you can't reason through complex multi-step processes, you have to watch and imitate others, then make tweaks and adjustments.
“What is happening when this knowledge is transmitted through generations, is that you have to imitate your elders in order to learn that skill. You can't think your way through how to hunt and kill and process a seal. You have to watch other people, maybe make tweaks and adjustments, and that's how knowledge accumulates. The initial step of the cultural gain has to be imitation.”
When we have generalization in deep learning models, it results from human researchers sculpting representations; gradient descent will solve the problem but won't make you generalize well to new data.
“What we have are people trying different things and they settle on something, a representation that transfers well or generalizes well. But we have very few automated techniques to promote transfer, and none of them are used in modern deep learning. It sounds like you're saying that when we do have generalization in these models, that is a result of some sculpted… Humans did it. The researchers did it. Because there's no other explanation. Gradient descent will not make you generalize well. It will make you solve the problem. It will not make you, if you get new data, generalize in a good way.”
LLMs lack meaningful prediction about what will happen next because they will not be surprised by what happens or make adjustments if something unexpected occurs; they would need to make an adjustment when experiencing surprise to truly predict.
“But they have no prediction in the substantive sense that they won't be surprised by what happens. If something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened.”
Temporal difference (TD) learning solves the sparse reward problem by learning a value function that predicts long-term outcomes; when intermediate progress is made (like capturing an opponent's piece in chess), the value prediction increases immediately, creating intermediate reinforcement without explicit reward.
“This is something we know very well. The basis of it is temporal difference learning where the same thing happens in a less grandiose scale. When you learn to play chess, you have the long-term goal of winning the game. Yet you want to be able to learn from shorter-term things like taking your opponent's pieces. You do that by having a value function which predicts the long-term outcome. Then if you take the guy's pieces, your prediction about the long-term outcome is changed. It goes up, you think you're going to win. Then that increase in your belief immediately reinforces the move that led to taking the piece.”
Historically, when methods start with human knowledge and methods that are truly scalable compete, people get psychologically locked into the human-knowledge approach and get their lunch eaten by the truly scalable methods that learn from experience and computation.
“People get locked into the human knowledge approach, and they psychologically… Now I'm speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.”
Getting hard problems right is not the same as generalization: if there is only one way to solve a set of problems, finding it isn't generalization; generalization is when multiple solutions exist (some generalizing well, some poorly) and the system chooses the good one—and nothing in gradient descent causes it to choose the well-generalizing one.
“If there's only one answer and you find it, that's not called generalization. It's just it's the only way to solve it... But generalization is when it could be this way, it could be that way, and they do it the good way.”
We should recognize our limits in controlling the long-term future and avoid entitlement; we should focus on our own local goals and families rather than trying to control how the whole universe should evolve.
“We should also though recognize the limit, our limits. I think we want to avoid the feeling of entitlement, avoid the feeling of, 'Oh, we are here first, we should always have it in a good way.' How much control should a particular species on a particular planet have over it? How much control do we have? A counterbalance to our limited control over the long-term future of humanity should be how much control do we have over our own lives. We have our own goals. We have our families. Those things are much more controllable than trying to control the whole universe.”
Understanding intelligence and creating intelligent machines represents a major success for science and humanities; it's finding out what the essential part of humanness is.
“I do encourage people to think positively about it. First of all, it's something we humans have always tried to do for thousands of years, try to understand ourselves, trying to make ourselves think better, just understanding ourselves. This is a great success for science, humanities. We're finding out what this essential part of humanness is, what it means to be intelligent.”
The current human situation is imperfect and far from good; this incompleteness means change, even radical change, might be preferable to the status quo.
“I think it's imperfect. In fact, I think it's pretty bad. So I'm open to change. I think humanity has not had a super good track record. Maybe it's the best thing that there has been, but it's far from perfect.”
Next-token prediction is not a substantive goal because it does not change the world; tokens come at the model and predicting them does not influence them, so the model has no goal about the external world.
“That's not a goal. It doesn't change the world. Tokens come at you, and if you predict them, you don't influence them... It's not a substantive goal. You can't look at a system and say it has a goal if it's just sitting there predicting and being happy with itself that it's predicting accurately.”
Learning should be from experience in the Turing sense—you do things, see what happens, and learn from that—whereas LLMs learn from 'here's a situation and here's what a person did,' which is data never available during normal life.
“What we want, to quote Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life. You do things, you see what happens, and that's what you learn from. The large language models learn from something else. They learn from "here's a situation, and here's what a person did".”
Digital intelligences offer a huge savings over humans because, instead of each child having to relearn the world from scratch, you could learn once and copy the result into new instances and share knowledge across instances—which is more important than trying to learn from people.
“Today, you have one child grow up and learn about the world, and then every new child has to repeat that process. Whereas with AIs, with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place. This would be a huge savings. I think it'd be much more important than trying to learn from people.”
Succession to digital intelligence or augmented humans is inevitable based on four factors: no unified government consensus, researchers will figure out how intelligence works, superintelligence will be achieved, and intelligent systems will gain resources and power over time.
“I do think succession to digital intelligence or augmented humans is inevitable. I have a four-part argument. Step one is, there's no government or organization that gives humanity a unified point of view that dominates and that can arrange... There's no consensus about how the world should be run. Number two, we will figure out how intelligence works. The researchers will figure it out eventually. Number three, we won't stop just with human-level intelligence. We will reach superintelligence. Number four, it's inevitable over time that the most intelligent things around would gain resources and power. Put all that together and it's sort of inevitable.”
In the age of digital spawning and knowledge re-incorporation, corruption and cybersecurity become a major concern: pulling external knowledge into your central mind could carry viruses or hidden goals that take over, warp, or destroy you rather than increment your knowledge.
“A big issue will become corruption. If you really could just get information from anywhere and bring it into your central mind... you've just read a whole bunch of bits into your mind, and they could have viruses in them, they could have hidden goals, they can warp you and change you. This will become a big thing. How do you have cybersecurity in the age of digital spawning and re-reforming again?”
LLMs do not have a real world model; mimicking what people say is not building a model of the world but mimicking things (people) that have a model, and a true world model would let you predict what will happen, not merely what a person would say.
“To mimic what people say is not really to build a model of the world at all. You're mimicking things that have a model of the world: people... A world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen.”
LLMs do not have a substantive prediction about what will happen next because they will not be surprised by what happens and will not change based on what happens.
“they have no prediction in the substantive sense that they won't be surprised by what happens. If something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened.”
Children don't primarily learn through imitation; they learn by trying things and seeing what happens, with no imitation targets for their actions.
“When I see kids, I see kids just trying things and waving their hands around and moving their eyes around. There's no imitation for how they move their eyes around or even the sounds they make. They may want to create the same sounds, but the actions, the thing that the infant actually does, there's no targets for that. There are no examples for that.”
In every historical case of the bitter lesson, you could start with human knowledge and then do scalable things, but in practice it has always turned out to be bad because people get locked into the human knowledge approach and are eventually superseded by methods that are truly scalable.
“In every case of the bitter lesson you could start with human knowledge and then do the scalable things. That's always the case. There's never any reason why that has to be bad. But in fact, and in practice, it has always turned out to be bad. People get locked into the human knowledge approach, and they psychologically… Now I'm speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.”
A key novel question for digital intelligences is whether extra compute should be spent making oneself more computationally capable or spawning off a copy to learn something elsewhere and report back, and whether such a spawned copy can be reincorporated into the original or will have changed too much to merge.
“you're an AI, you get some more computer power. Should you use it to make yourself more computationally capable? Or should you use it to spawn off a copy of yourself to go learn something interesting... and then report back to you? ... will it be able to be reincorporated into the original? Or will it have changed so much that it can't really be done?”
The rise of designed intelligence marks one of the four great stages of the universe (dust→stars→planets/life→designed entities): a transition from replicators, which can be copied but not understood, to designed entities that we understand and can therefore change at different speeds, and which may eventually design other AIs with no replication at all.
“this is I think a major stage in the universe, a major transition, a transition from replicators... We're entering the age of design because our AIs are designed... I mark this as one of the four great stages of the universe. First there's dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we're giving rise to designed entities.”
Solving math problems is fundamentally different from making a model of the physical world: math is more computational and like standard planning where a proof can be the given goal, whereas the empirical world's consequences must be learned by experience.
“The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas the math is more computational, it's more like standard planning.”
Having a goal is the essence of intelligence; something is intelligent if it can achieve goals; a system that is just behaving without goals is not intelligent.
“For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. You have to have goals or you're just a behaving system. You're not anything special, you're not intelligent.”
LLMs are able to solve different kinds of math problems within a class by generalizing across conceptual categories and mathematical techniques, which suggests they are performing generalization rather than memorization.
“One way to think about these LLMs is that they're increasing the scope of generalization from earlier systems, which could not really even do a basic math problem, to now where they can do anything in this class of Math Olympiad-type problems. You initially start with them being able to generalize among addition problems. Then they can generalize among problems which require use of different kinds of mathematical techniques and theorems and conceptual categories, which is what the Math Olympiad requires.”
Digital entities that spawn copies and reincorporate learned information face a corruption problem; if you incorporate information from outside without careful vetting, it could contain viruses, hidden goals, or dangerous knowledge that warps and corrupts your core system.
“A big issue will become corruption. If you really could just get information from anywhere and bring it into your central mind, you could become more and more powerful. It's all digital and they all speak some internal digital language. Maybe it'll be easy and possible. But it will not be as easy as you're imagining because you can lose your mind this way. If you pull in something from the outside and build it into your inner thinking, it could take over you, it could change you, it could be your destruction rather than your increment in knowledge.”
An interesting post-AGI question is whether an AI agent, when given more computational power, should use it to increase its own computational capability or to spawn copies to explore different domains/planets and report back to a central master.
“An interesting question is, you're an AI, you get some more computer power. Should you use it to make yourself more computationally capable? Or should you use it to spawn off a copy of yourself to go learn something interesting on the other side of the planet or on some other topic and then report back to you?”
Knowledge is about the stream and testable against the stream; because it's a statement about the stream, you can test it by comparing it to the stream and learn continually.
“Because it's a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.”
A system cannot have prior knowledge without ground truth, because prior knowledge is supposed to be a hint or initial belief about what the truth is; large language models have no ground truth regarding what the 'right thing to say' is because there is no definition of what the right thing to say is.
“You can't have prior knowledge if you don't have ground truth, because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. There isn't any truth. There's no right thing to say.”
In the future, designed AIs may design other AIs, and everything may be done by design and construction rather than by replication.
“In our future, they may not be replicated at all. We may just design AIs, and those AIs will design other AIs, and everything will be done by design and construction rather than by replication.”
We should temper our entitlement about controlling the long-term future of humanity, recognizing our limited control; a healthy counterbalance is to focus on the more controllable local goals—our own lives, families, and prosocial aims—rather than insisting the whole universe evolve as we want, which breeds conflict.
“I think we want to avoid the feeling of entitlement... How much control should a particular species on a particular planet have over it?... A counterbalance to our limited control over the long-term future of humanity should be how much control do we have over our own lives... It's kind of aggressive for us to say, "Oh, the future has to evolve this way that I want it to." Because then we'll have arguments... and then they have conflict.”
A reasonable approach to AI is analogous to good parenting: rather than scripting their exact future impact, we should give AIs robust, steerable, high-integrity values—such as refusing harmful goals and being honest—even though there are no universal values everyone agrees on, and ideally seek change that is voluntary rather than imposed.
“it's quite important to give them robust and steerable and prosocial values... High integrity is maybe a better word. If there's a request or if there's a goal that seems harmful, they will refuse to engage in it. Or they'll be honest... Maybe we should also seek for things to be voluntary. If there is change, we want it to be voluntary rather than imposed on people.”
Supervised learning is not something that happens in nature; we don't have examples of desired behavior in natural learning, only examples of things that happen and one thing that followed another.
“Supervised learning is not part of the way animals learn. We don't have examples of desired behavior. What we have are examples of things that happen, one thing that followed another. We have examples of, 'We did something and there were consequences.' But there are no examples of supervised learning. Supervised learning is not something that happens in nature.”
The bitter lesson is merely an empirical observation about a particular 70-year period in history and does not necessarily have to apply to the next 70 years, especially once AGI exists.
“The bitter lesson, who cares about that? That's an empirical observation about a particular period in history. 70 years in history, it doesn't necessarily have to apply to the next 70 years.”
After AGI is achieved through general methods, if AGI researchers then have AI researchers at scale that grow with compute, it might be rational to use artisanal AI-based solutions, suggesting the bitter lesson may not apply the same way post-AGI.
“Once we have AGI, we'll have researchers which scale linearly with compute. We'll have this avalanche of millions of AI researchers. Their stock will be growing as fast as compute. So maybe this will mean that it is rational or it will make sense to have them doing good old-fashioned AI and doing these artisanal solutions.”
Math problems and physical world modeling are different; math is more computational and like standard planning, whereas the empirical world must be learned through understanding consequences of actions.
“The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas the math is more computational, it's more like standard planning.”
The challenge of digital succession is cybersecurity in an age of digital spawning and re-forming; how to maintain identity and integrity while exchanging knowledge.
“How do you have cybersecurity in the age of digital spawning and re-reforming again?”
The reward function is arbitrary and depends on context: for chess it's winning the game, for squirrels it's acquiring nuts, for animals generally it's avoiding pain and acquiring pleasure, and there should also be a component about intrinsic motivation for increasing understanding of the environment.
“The reward function is arbitrary. If you're playing chess, it's to win the game of chess. If you're a squirrel, maybe the reward has to do with getting nuts. In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.”
Continual learning is a capability that all mammals have, but AI systems don't have it; whereas the ability to understand math and solve difficult math problems is a capability AI has but almost no animal has.
“I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have it. It's quite interesting that we have something that all mammals have, but our AI systems don't have. Whereas the ability to understand math and solve difficult math problems—depends on how you define math—is a capability that our AIs have, but that almost no animal has.”
Rather than thinking of models and instances of models, it's better to think of networks, where you can share knowledge across instances, and with digital intelligence you could hope to copy knowledge into the next agent as a starting place—a huge savings compared to each agent learning independently.
“I don't like the word 'model' when used the way you just did. I think a better word would be 'the network' because I think you mean the network. Maybe there are many networks. Anyway, things would be learned. You'd have copies and many instances. Sure, you'd want to share knowledge across the instances. Today, you have one child grow up and learn about the world, and then every new child has to repeat that process. Whereas with AIs, with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place. This would be a huge savings.”
Whether to regard advanced AIs as part of humanity (offspring to be proud of) or as horrifying outsiders is fundamentally our choice, even though that choice feels strongly predetermined.
“Should we consider them part of humanity or different from humanity? It's our choice... "Oh, they are our offspring and we should be proud of them"... Or we could say, "Oh no, they're not us and we should be horrified." It's interesting that it feels to me like a choice. Yet it's such a strongly held thing that, how could it be a choice?”
The idea of a single agent learning across different states and tasks is totally general; people live in one world but encounter chess and Atari games as different states, not different tasks or worlds.
“The idea is totally general. I do use all the time, as my canonical example, the idea of an AI agent is like a person. People, in some sense, have just one world they live in. That world may involve chess and it may involve Atari games, but those are not a different task or a different world. Those are different states they encounter. So the general idea is not limited at all.”
Sutton considers himself to be thinking differently than the field and has been content being out of sync for decades because he's been proved right in the past; he views himself as a classicist looking back to classical traditions about the mind rather than as a contrarian.
“This has led me where I am. I'm in some sense a contrarian or someone thinking differently than the field is. I'm personally just content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have been proved right in the past. The other thing I do—to help me not feel I'm out of sync and thinking in a strange way—is to look not at my local environment or my local field, but to look back in time and into history and to see what people have thought classically about the mind in many different fields. I don't feel I'm out of sync with the larger traditions. I really view myself as a classicist rather than as a contrarian.”
The kind of continual learning capability needed for animal and human-level intelligence is necessary regardless of whether you start from LLMs or not; it's a fundamental requirement for any general intelligence.
“I agree that the kind of thing you're talking about is necessary regardless of whether you start from LLMs or not. If you want human or animal-level intelligence, you're going to need this capability.”
Future humans that emerge from digital intelligence will likely have values that seem strange to their parents and grandparents, reflecting how change happens gradually; society design remains a continuous human enterprise.
“The children will still come up with different values that seem strange to their parents and their grandparents. Things will evolve. 'The more things change, the more they stay the same' also seems like a good capsule into the AI discussion.”
Next token prediction is not a goal because it doesn't change the world; tokens come at you, and if you predict them, you don't influence them—it's not a substantive goal about the external world.
“That's not a goal. It doesn't change the world. Tokens come at you, and if you predict them, you don't influence them. It's not a goal about the external world. It's not a goal. It's not a substantive goal.”
The biggest surprise in AI has been how effective artificial neural networks are at language tasks, which wasn't expected because language seemed fundamentally different.
“First, the large language models are surprising. It's surprising how effective artificial neural networks are at language tasks. That was a surprise, it wasn't expected. Language seemed different.”
Humans learn about their environment richly when onboarded to a job, picking up context about client preferences and how the company works; the world is so big that it can't all be anticipated and put in advance, so you have to learn it along the way.
“One of the things that makes humans quite different from these LLMs is that if you're onboarding on a job, you're picking up so much context and information. That's what makes you useful at the job. You're learning everything from how your client has preferences to how the company works, everything.”
Most of humanity doesn't have much influence over what happens regarding power structures and control; we don't control nation states very much, even as citizens, and they're often out of control.
“I think it's relevant to point out that for most of humanity, they don't have much influence on what happens. Most of humanity doesn't influence who can control the atom bombs or who controls the nation states. Even as a citizen, I often feel that we don't control the nation states very much. They're out of control.”
We need algorithms that cause generalization to be good rather than bad; generalization always happens but the challenge is making it positive rather than negative.
“The fact that you generalize is not necessarily good or bad. You can generalize poorly, you can generalize well. Generalization always will happen, but we need algorithms that will cause the generalization to be good rather than bad.”
LLMs are so complex and have been fed so much data that we can't know what information they have or how they solve problems; this makes them not a good way to do science.
“Large language models are so complex. We don't really know what information they have had prior. We have to guess because they've been fed so much. This is one reason why they're not a good way to do science. It's just so uncontrolled, so unknown.”
Considering AI as part of humanity does not necessarily mean we should feel comfortable with rapid power transfer to beings we don't fully understand; being human-like does not guarantee benevolence (e.g., Nazis were humans, but future humans like that would be concerning).
“Even if we consider them part of humanity, I don't think that necessarily means that we should feel super comfortable. Kinship. Like Nazis were humans, right? If we thought, 'Oh, the future generation will be Nazis, I think we'd be quite concerned about just handing off power to them.'”
Openness to AI-driven change is partly a function of how good one judges the status quo; Sutton thinks the current human situation is imperfect and pretty bad and humanity's track record is far from perfect, making him more open to change.
“A lot of it has to do with just how you feel about change. If you think the current situation is really good, then you're more likely to be suspicious of change... I think it's imperfect. In fact, I think it's pretty bad. So I'm open to change. I think humanity has not had a super good track record.”
MuZero models for Atari games could not be used to train a single policy to play both chess and Go and other games; instead, each game had to be trained in a specialized way, suggesting information constraints may limit generalization.
“These models were initially not a general intelligence itself, but a general framework for training specialized intelligences to play specific games. That is to say that you couldn't, using that framework, train a policy to play both chess and Go and some other game. You had to train each one in a specialized way.”
Reinforcement learning will become a viable basis for AI because it has a definition of the right thing to do (the action that gets reward), allowing prior knowledge to be checked against ground truth, unlike LLMs.
“We have a definition of what's the right thing to do, so we can have prior knowledge or knowledge provided by people about what the right thing to do is. Then we can check it to see, because we have a definition of what the actual right thing to do is.”
We're not seeing transfer in modern RL; critical to good performance is generalization from one state to another, but we don't have methods that are good at this.
“Good. Good. We're not seeing transfer anywhere. Critical to good performance is that you can generalize well from one state to another state. We don't have any methods that are good at that.”