What this covers
Dwarkesh Patel responds to Rich Sutton's argument that large language models represent a dead-end path to AGI, separated by a fundamental divide from true reinforcement learning-based intelligence. Patel contests this dichotomy directly: he argues that imitation learning is not categorically distinct from RL but rather short-horizon RL, and that the knowledge LLMs absorb from human data—far from being a cul-de-sac—can serve as a prior that bootstraps and accelerates learning from ground truth. The core claim is that imitation and experiential learning are continuous and complementary rather than opposing paradigms, and that modeling humans can yield representations robust enough to ground real-world learning.
Patel's response spans several technical and conceptual fronts. He grants that Sutton identifies genuine gaps: continual learning, sample efficiency, and dependence on finite human data are real constraints on current systems. But he disputes whether these gaps vindicate Sutton's vision or merely describe features of today's training regime. He notes that models already develop deep, flexible world models despite training only on human data, and that in-context learning demonstrates something resembling continual learning within existing architectures. On the path forward, Patel sketches how continual learning might be grafted onto LLMs via test-time fine-tuning or meta-learning across longer horizons. He draws an analogy to energy transition: pretraining data, like fossil fuels, is non-renewable but may remain the necessary bridge to post-scarcity learning, and he invokes historical precedent—AlphaGo and AlphaZero—to show that human priors need not prevent superhuman performance. The disagreement is not whether the gaps Sutton names exist, but whether closing them requires abandoning what LLMs have learned.
Patel argues that Sutton's first-principles dichotomy between imitation learning (LLMs) and 'true' experiential intelligence is overstated: imitation learning is continuous with and complementary to RL, can yield real world models, and likely provides the prior needed to bootstrap AGI even if a from-scratch paradigm eventually wins.
- Imitation learning is just short-horizon RL and provides a prior that kickstarts ground-truth RL
- LLMs already develop deep world representations and in-context learning despite training only on human data
- Continual learning gaps are real but may be shoehorned atop LLMs via test-time fine-tuning
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Humanity's success rests on accumulating knowledge over tens of thousands of years, inheriting language, legal systems, and most technological knowledge from prior generations, a process more analogous to imitation learning than to RL from scratch.
“We didn’t invent the language we speak, nor the legal system we use, nor even most of the knowledge relevant to the technologies in our phones. This process is more analogous to imitation learning than to RL from scratch.”
Imitation learning is not categorically different from RL; it is just short-horizon RL where the episode is a single token long: the LLM conjectures the next token from its world understanding and receives reward proportional to how well it predicted the true token.
“Imitation learning is just short horizon RL. The episode is a token long. The LLM makes a conjecture about the next token based on its understanding of the world and the other information in the sequence. And it receives reward in proportion to how well it predicted the true token.”
Even if Sutton's Platonic ideal is not the path to the first AGI, he identifies genuine basic gaps that go unnoticed because they pervade the current paradigm—lack of continual learning, abysmal sample efficiency, and dependence on exhaustible human data—and if LLMs reach AGI first, their successor systems will almost certainly be based on Sutton's vision.
“he’s identifying genuine basic gaps which we don’t even notice because they are so pervasive in the current paradigm: lack of continual learning, abysmal sample efficiency, dependence on exhaustible human data.”
LLMs do not build a true world model that predicts how the environment changes in response to actions; instead they model what a human would say next, relying on human-derived concepts, so an LLM trained on 1900 data could not derive relativity from scratch.
“Rather, they are building a model of what a human would say next. And this leads them to rely on human-derived concepts. If you trained an LLM on the data from 1900, it wouldn’t be able to come up with relativity from scratch.”
Even the RLVR done on LLMs relies on human-furnished RL environments designed to teach prescribed skills, so the agent is in no substantial way learning from organic, self-directed engagement with the world, and learning only from inelastic human data is not a scalable use of compute.
“these RL environments are human furnished playgrounds to teach LLMs the specific skills we have prescribed for them.”
Imitation learning can leverage models to learn better from ground truth: models have been RLed to win IMO Gold and code entire applications from scratch—ground-truth examinations—but you couldn't RL a model to do these from scratch; you needed a reasonable prior over human data to kickstart the RL process.
“We have RLed models to win Gold in IMO and code up entire working applications from scratch. These are “ground truth” examinations. But you couldn’t RL a model to accomplish these feats from scratch. You needed a reasonable prior over human data in order to kick start the RL process.”
An LLM being RLed on outcome-based rewards learns only O(1) bits per episode even though the episode may be tens of thousands of tokens long, whereas animals extract far more information from interacting with their environment than just the end-of-episode reward signal.
“An LLM being RLed on outcome-based rewards learns O(1) bits per episode, and the episode may be tens of thousands of tokens long.”
Patel's main disagreement with Sutton is that the concepts Sutton uses to distinguish LLMs from true intelligence are not actually mutually exclusive and dichotomous: imitation learning is continuous with and complementary to RL, and models of humans can provide a prior that facilitates learning true world models.
“My main difference with Rich is that I think the concepts he’s using to distinguish LLMs from true intelligence are not actually mutually exclusive and dichotomous.”
Because LLMs cannot learn on-the-job, a new architecture enabling continual learning will be needed, and once it exists the agent will learn on-the-fly like humans and animals, with no special training phase, rendering the current LLM approach obsolete.
“LLMs aren’t capable of learning on-the-job, so we’ll need some new architecture to enable continual learning.”
The bitter lesson is not about throwing away as much compute as possible, but about developing techniques that most effectively and scalably leverage compute.
“The bitter lesson says that you want to come up with techniques which most effectively and scalably leverage compute.”
Being able to continuously learn from the environment in a high-throughput way is necessary for true AGI and clearly does not exist with LLMs trained on RLVR.
“Being able to continuously learn from the environment in a high throughput way is obviously necessary for true AGI. And it clearly doesn’t exist with LLMs trained on RLVR.”
Whether the prior obtained from human data is a proper 'world model' or just a model of humans is a semantic debate, because what matters is whether modeling humans helps the system start learning from ground truth and become a true world model; defining 'world model' by the process needed to build one rather than by the capabilities it implies is mistaken given LLMs have clearly developed deep, flexible, coherent representations of the world.
“Whether you wanna call this prior a proper “world model”, or just a model of humans, seems like a semantic debate to be honest.”
AlphaGo (conditioned on human games) and AlphaZero (bootstrapped from scratch) were both superhuman, with AlphaZero better; this shows human data isn't necessarily detrimental—at enough scale it just stops being significantly helpful—so imitation learning can still play a role in developing the first AGI or ASI.
“AlphaGo was still superhuman, despite being initially shepherded by human player data. The human data isn’t necessarily actively detrimental - at enough scale it just isn’t significantly helpful.”
Evolution does meta-RL to make an RL agent that can selectively do imitation learning, whereas with LLMs we go the opposite way: first building a base model that does pure imitation learning, then doing RL on it to make it a coherent agent with goals and self-awareness.
“Evolution does meta-RL to make an RL agent. That agent can selectively do imitation learning. With LLMs, we’re going the opposite way. We first made a base model that does pure imitation learning. Then we do RL on it to make it a coherent agent with goals and self-awareness.”
Most compute spent on an LLM is used during deployment, yet the model learns nothing during that time, learning only during a special training phase, which is not an effective use of compute.
“Most of the compute spent on an LLM is used on running it in deployment. And yet it’s not learning anything during this time!”
Models already demonstrate something resembling human continual learning within their context windows, and because in-context learning emerged spontaneously from the training incentive to process long sequences, if information could flow across windows longer than the current context limit, models could meta-learn the same flexibility they already show in-context.
“The fact that in-context learning emerged spontaneously from the training incentive to process long sequences suggests that if information could flow across windows longer than the current context limit, models could meta-learn the same flexibility they already show in-context.”
Continual learning might be shoehorned atop LLMs by making supervised fine-tuning a tool call for the model, so the outer-loop RL incentivizes the model to teach itself using supervised learning to solve problems that don't fit in the context window.
“one could imagine making SFT a tool call for the model. So the outer loop RL is incentivizing the model to teach itself effectively using supervised learning, in order to solve problems that don’t fit in the context window.”
Pretraining data is analogous to fossil fuels: just because it is non-renewable does not mean civilization is on a dead-end track, because you couldn't have transitioned from water wheels in 1800 directly to solar and fusion without a cheap, convenient, plentiful intermediary.
“Ilya compared pretraining data to fossil fuels. This analogy has remarkable reach. Just because fossil fuels are not renewable does not mean that our civilization ended up on a dead-end track by using them.”
Human cultural learning is not literally next-token prediction nor pursuit of a well-defined scalar reward; no ML learning regime perfectly describes human learning, which is analogous to both RL and supervised learning—as planes are to birds, supervised learning might be to human cultural learning.
“No ML learning regime perfectly describes human learning. We do things which are analogous to both RL and supervised learning. As planes are to birds, supervised learning might be to human cultural learning.”
If you tried to fit the transition-model spec into LLMs by fine-tuning on all observed tokens, in practice the most naive way of doing this doesn't work well, according to researcher friends.
“From what I hear from my researcher friends, in practice the most naive way of doing this doesn’t work well.”