Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job
What this covers
Dwarkesh Patel talks alone, working through what the labs' current recipe leaves out. He starts from RLVR and asks what makes a domain tractable. His answer adds a second condition, being grindable: cheap parallel rollouts against a deterministic, replayable simulator. He cites slow progress on computer use as evidence. Building a business, winning court cases, day trading, and running a campaign can't be recreated in a datacenter. He treats RLVR's generalization as an open empirical question, and reads a remark by Dario Amodei about context length as a possible sign that short-horizon training doesn't carry over to long horizons, while admitting he may be over-reading it.
The middle section turns to mechanics. He questions how sample efficiency should be measured, using EfficientZero on Atari. He contrasts how information is stored in context and in weights: roughly 320 KB of KV cache per token against about 0.075 bits per token in pretraining. He also argues that human compression into intuition aids generalization, while perfect recall can cripple abstraction. Shipped online learners like Cursor Tab, he notes, learn one shared objective across hundreds of millions of daily requests rather than learning different things per user.
The proposals come last. On-Policy Self-Distillation trains the base model to predict what a context-equipped copy would predict. He argues it needs no outer-loop reward and supplies denser supervision than RLVR, though only until the student first errs, which he thinks Trajectory-Refined Distillation could fix. He then floats dreaming, where a model builds simulations to rehearse skills, as a possible fourth scaling axis. He also concedes that longer context might make weight updates unnecessary.
Patel argues the labs' bet that RL on verifiable tasks will yield AGI is incomplete: domains that are verifiable but not grindable progress slowly, and real-world skills need sample-efficient continual learning that writes deployment experience back into the weights.
- Verifiability alone is insufficient; a domain also needs cheap parallel rollouts against a deterministic, replayable simulator.
- Real-world domains like business, law, and politics have sparse, non-replayable feedback, so they demand sample efficiency that current RLVR does not provide.
- Learning must move from the context window into weights via methods such as on-policy self-distillation (OPSD) and possibly 'dreaming', because in-context learning cannot accumulate across deployments.
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
EfficientZero, trained against an Atari simulator, might beat a novice human in two hours, but it plays dozens of simulated games in its head per real step, so whether it counts as more sample efficient depends on how sample efficiency is measured.
“Because for each step in the real game, EfficientZero is playing dozens of simulated games in its head.”
Human learning compresses observations into intuitions and big-picture knowledge in the weights, which aids generalization and grokking, whereas perfect-fidelity recall of raw information, as in some savants, can cripple the ability to understand abstractions and metaphors.
“such sheer volume cripples these humans' ability to understand abstractions and metaphors. Human continual learning is less about having all your observations at the tip of your tongue, and more about chiseling the right intuitions and big picture knowledge back into the weights.”
For Llama 3 70B, the KV cache grows about 320 KB per token of in-context learning, whereas pretraining stores about 0.075 bits per token, a roughly 35-million-fold difference in information stored per token between in-context learning and pretraining.
“The KV cache (aka the representation that is built up from learning the context) grows 320 KB with each token. Whereas in training, the model only stores 0.075 bits per token”
On-policy self-distillation (OPSD) distills a session into weights by encouraging the base model to make the same predictions on real-world problems that a model with the full accumulated session context would make, so the base model learns what the context-equipped teacher knows.
“the idea is that we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made.”
Whether RLVR generalizes to all domains is an open empirical question; scaling RL environment spending from billions to a trillion dollars would not guarantee a fully general, human-like intelligence operating within the context window.
“Whether RLVR generalizes that well is an open empirical question: if labs went from spending billions of dollars on RL environments to a trillion dollars, would you get a fully general, human-like intelligence operating within the context window?”
Shipped online-learning models such as the Cursor Tab model learn the same objective across over 400 million requests a day, namely which edits users accept, and so far no model has been seen learning different things for different users.
“the Cursor Tab model online-learns by predicting the same exact objective for over 400M+ requests a day (that objective is which edits got accepted)”
RL concentrates gradient updates only on what is relevant to getting the outcome right, making its updates sparse, which matters for continual learning because on-the-job learning must not overwrite everything else the base model knows.
“RL training doesn't suffer from this failure mode, and it's great at concentrating the gradient update to only what is relevant to getting the outcome right”
Dario Amodei's remark that performance degrades when a model trained at a short context length is served at a long one suggests short-horizon RL training may not generalize to long-horizon performance, though Patel admits he may be over-reading it.
“If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.”
Continual learning, defined as weights updating from deployment, may not be necessary if in-context learning becomes good enough across longer horizons, since six months of onboarding could in principle fit into an arbitrarily large context window.
“But what if you could just fit those six months into the context window? There's been tons of architectural innovations on the transformer which dramatically increase the length of context you can store.”
A domain must be not only verifiable but also grindable, meaning you can run many parallel rollouts against a deterministic and replayable simulator; verifiability alone does not make a domain amenable to RL progress.
“it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator.”
The most valuable information a model could learn from is revealed only in deployment, such as what organizations are actually doing with the model, what they use it for, and what mistakes it tends to make in the real world.
“it is only in deployment that the most valuable bits of information which your model could learn from are revealed”
Many skills an AGI would need, such as building a business, winning court cases, day trading, or running an election campaign, cannot be recreated in a datacenter because rollouts require real-world interaction and outer-loop verification can take months or years.
“The rollout requires interacting with the world and cannot be recreated simply within the datacenter. And the outer loop verification may take months or years of real world actions to elicit”
By the time continual learning matures, the main way AI improves will be accumulated experience from broad deployment rather than pre-release training, so every interaction makes the AI smarter for all users, which Patel calls scary and exciting.
“the main way that AI gets better is not through the training received before the model is released to the public. Rather, it's from all this experience that they are accumulating from being broadly deployed”
If an AI can build a good simulation of reality to rehearse new skills and try alternative strategies, it could experience orders of magnitude more simulated samples in the same wall-clock time, a speculative idea Patel calls dreaming.
“If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what works, then it could experience orders of magnitude more simulated samples in the same wall clock time.”
The slow progress of computer use shows that unless a domain has a very replayable training target, models will struggle to make much progress, because models are highly sample inefficient during training.
“its current lethargy tells us the following: that unless you can build a very replayable training target for a domain, the models will struggle to make much progress. The reason this is true is, of course, that the models are incredibly sample inefficient during training.”
Moving learning into the weights sacrifices the sample efficiency of in-context learning, because gradient updates are extremely sample inefficient, which is why shipped online-learning models have all learned the same thing across millions of users.
“But the moment you move into the weights, you have to give up on in-context learning's sample efficiency. Because gradient updates are super sample-inefficient”
Sample efficiency and continual learning are deeply connected because on-the-job data is scarce and learning from it requires sample efficiency; in-context fast weights from attention scale poorly in memory, so architectures need an intermediate representation.
“the 'fast weights' built on the fly by attention which allow for this sample efficiency scale very poorly in terms of memory. So we need architectural innovations which allow for some kind of intermediate representation.”
RL learns less information per sample than supervised learning, but this may be beneficial because the model changes only as much as necessary to achieve the outcome, and OPSD preserves that property.
“RL learns much less information per sample than supervised learning. But this may be a good thing rather than a bad thing”
Around 30 to 50 percent of a lab's compute goes to inference, and that compute currently does nothing to improve the model, which Patel calls a waste.
“Around 30-50% of a lab's compute goes to inference, and that compute is currently not really doing anything productive in helping improve the model. What a waste!”
If dreaming works, it would become a fourth axis of scaling alongside pretraining, RL, and inference-time compute, with the model spending compute to write RL environments that rehearse skills needed for a specific user's production work.
“If it works, it would become a fourth axis of scaling, alongside pretraining, RL, and inference-time compute.”
After RLVR makes an agent competent enough to deploy, continual learning can improve it in domains adjacent to its pre-deployment training, and each round extends its skill frontier, so capability expands beyond the verifiable domains it was trained on.
“AI can get better at domains that are adjacent to what it was explicitly trained for beforehand with RLVR. And in the next round it gets better at the thing adjacent to what it was previously online learned.”
Each job, company, and problem is different, so AIs need to learn deployment-specific information such as how an organization works and fits together, which cannot be stuffed into a shared training run.
“each job and company and problem is different, and you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can't be stuffed into a shared training run.”
OPSD is better than RLVR for two reasons: it does not require an outer-loop verifiable reward, only a model that can learn the right things within the context window, and it provides much denser supervision by training on per-token probability discrepancies between teacher and student instead of projecting one reward across the whole trajectory.
“OPSD doesn't require an outer loop verifiable reward. We just need a model that can learn the right things within the context window.”
OPSD gives dense supervision only up to the point the student first errs, after which the teacher stops giving useful feedback on an already-mistaken trajectory; this seems fixable by Trajectory-Refined Distillation, in which the teacher rewrites the trajectory from the error onward into a correct continuation.
“This seems fixable by a technique called Trajectory-Refined Distillation, where the teacher rewrites the trajectory from the error onward into a complete, correct continuation.”
For AIs to develop all human skills and skills no human has, they must learn from unstructured, unverifiable, and ambiguous information gathered from scarce real-world interaction, because in many domains the relevant training information exists in no other form.
“they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real world interaction. Because in many domains, the relevant training information simply doesn't exist in any other way.”
Patel does not think agents can accumulate new skills by passing themselves notes, as with Markdown files, and believes real progress requires updating the weights; he illustrates this with students passing notes on how to play an unfamiliar saxophone.
“I just don't think you can accumulate new skills by passing yourself notes.”
AIs cannot learn continually by building up a growing KV cache across users, because that is not scalable and is not how humans learn; human learning compresses information into weights rather than storing parameters and activations separately.
“AIs can't just keep building up a KV cache that grows in size as you keep learning from more and more users. That's just not scalable, and it's also not how humans learn.”
By the end of 2027, an AI may be able to cowork with a user for a full week of wall-clock time, after which the user gives a thumbs up or down, and a thumbs up triggers the base model to distill the session's learnings using OPSD, dreaming, or other techniques.
“By this point, effective context lengths may have expanded such that this AI can cowork with you for a full week of wall clock time. At the end of the week you give it a thumbs up or a thumbs down.”
Architecture does not seem to be the fundamental bottleneck for continual learning, since many working ideas such as sparse attention and KV cache compaction already exist, so the loss function may be the real bottleneck.
“It doesn't seem to me that architecture is fundamentally the bottleneck to continual learning.”
Naively fine-tuning a base model to predict every token observed during a session is a nonsensical learning target, because people improve at jobs by consolidating a few relevant insights rather than recalling every transcript with perfect fidelity.
“But this makes no sense as a learning target - the way you get better at your job is not by recalling the transcript of what happened through every single day with perfect fidelity.”
Computer-use agents cannot be trained by having a thousand agents attempt the same checkout flow on a live site like Amazon, because the site detects and shuts down the bots.
“You can't have a thousand agents go try the same checkout flow on Amazon.com. Because Andy Jassy will find and detect your bots and shut your ass down.”
The workaround of building clones of Slack, Gmail, and other common applications for computer-use training is currently very labor-intensive and unscalable, though it could scale once AIs can code high-fidelity clones themselves.
“You can solve this by making clones of Slack, Gmail, and all the other common applications and websites. But at least currently, this is a very labor-intensive and unscalable way to build environments.”
Learning in reset-free, non-stationary environments is a known open problem in RL, and because data in most real-world domains is idiosyncratic and sparse, proficiency in those domains requires sample efficiency.
“Dealing with such reset-free non-stationary environments is a known open problem in RL. I'm not pointing out anything new. But I really do want to emphasize that because of the idiosyncratic and sparse nature of the data in most domains in the world, you need sample efficiency in order to get proficient.”
One hour of video consumes around 1 million tokens of text-equivalent context, which is one reason video data consumes the context window far faster than text and contributes to slower computer-use progress.
“Just one hour of video consumes around 1 million tokens of text.”
Dreaming is more speculative than game-playing precedents because building a simulation of the whole world is much harder than emulating the game of Go.
“The big difference, of course, is that it's much harder to build a simulation of the whole world than it is to emulate the game of Go.”
If short-horizon RL does not generalize to long horizons, then agents trained on white-collar tasks are unlikely to generalize to building a business from scratch as well as Sam Walton did.
“if we can't generalize from short to long horizon, how are agents supposed to generalize from lots of white collar task training”
Coding is easy to train with RL because one can create an environment with a software repo missing a feature and let a thousand parallel agents each work on an identical copy of the container.
“you can create an environment that has a software repo with some missing feature that you've tasked the AIs with creating, and then you have a thousand parallel agents just go at the problem, each with their identical copy of the container.”
AI agents are especially good at Go because it has a standardized package manager, whereas Python and TypeScript have a massive combinatorial space of frameworks, typing approaches, and utility libraries that is less amenable to clean, high-throughput parallel search via gradient descent.
“I've heard that AI agents are especially good at Go, because it has an excellent standardized package manager whereas Python and Typescript have a 'massive combinatorial space of frameworks, typing approaches, and utility libraries.'”
Having AIs rebuild complex applications from scratch is a strong RL objective for coding, and it doubles as a way to build the application clones needed for computer-use training.
“you're also killing two birds with one stone with this kind of procedure, because getting AIs to rebuild whole complex applications from scratch is a great RL objective for coding as well.”
Optimists believe fundamental deficits of the current paradigm, such as data inefficiency and lack of continual learning, can be steamrolled by scaling training, just as supposed fundamental NLP research problems collapsed under the compute thrown at LLMs.
“can be steamrolled by just scaling training more, just as all the supposed 'fundamental' research problems in natural language processing collapsed against the flood of compute thrown into LLMs”
The current research bet of the labs is that training AIs on millions of verifiable tasks across thousands of diverse RL environments will basically produce AGI, because such training builds general problem-solving skills like making sustained progress on open-ended tasks despite errors and ambiguity.
“if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we'll basically have built AGI”
Models are about one-millionth as sample efficient as humans during training, but this is a one-time cost amortized across billions of user sessions, so what matters is within-session sample efficiency, which has been improving with more RL training.
“Yes, these models are 1/1-millionth as sample efficient as humans during training. But training a one-time cost amortized across billions of user sessions.”
The current paradigm is like a genius grad student who never gets an internship: labs keep supplying classroom case studies through RL environments rather than letting the AI learn from the accumulated experience of broad deployment.
“We've got some genius grad student who has never been allowed to take an internship. And we keep giving it more and more classroom case studies in the form of RL training on environments.”
Hitting compact on a coding agent spends a small amount of compute to write a summary and gives a simulacrum of continual learning, whereas a /dream command would spend huge compute to build and train against a simulated version of what the model witnesses.
“Instead of hitting /compact on Codex or Cursor or Claude, which kindles a small amount of compute to write up a summary, and which gives you a simulacrum of continual learning, you hit /dream”
Patel asks what RL environment could make an AI as skilled at politics as Lyndon Johnson or at building a space launch business as Elon Musk, highlighting that no such environment obviously exists.
“What is the RL environment to make an AI as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk?”
With a couple more years of progress on transformer architectures that increase storable context length, arbitrarily large context windows may become achievable.
“There's been tons of architectural innovations on the transformer which dramatically increase the length of context you can store. With a couple more years of progress, why couldn't we have arbitrarily large context windows?”
The upcoming dreaming feature in the leaked Claude Code source is likely to be about the model writing lots of Markdown files for itself, whereas Patel means actually updating the weights.
“I'm not talking about the upcoming dreaming feature in the leaked Claude Code source code, which I'm guessing will be more about the model writing lots of Markdown files for itself.”