Fully autonomous robots are much closer than you think – Sergey Levine
What this covers
Sergey Levine speaks with Dwarkesh Patel about why general-purpose household robots are arriving sooner than the conventional timeline suggests—he estimates a median of about five years to functional home automation. The conversation centers on a single enabling shift: pretrained vision-language models now provide the prior knowledge and robust representations that robotics lacked five years ago. Rather than waiting for perfect lab solutions, Levine argues that deployment with humans in the loop creates a self-improvement flywheel, where robots learn continuously from corrected mistakes, language instruction, and feedback, much as coding assistants do. This differs fundamentally from earlier robotics paradigms, which treated learning as a phase to complete before deployment.
The conversation ranges across why manipulation tasks suit this learning-in-deployment model better than autonomous driving—mistakes can be corrected mid-task, yielding knowledge without catastrophe—and how hardware costs have plummeted as AI compensates for precision requirements. Levine discusses Moravec's Paradox to explain why robots can execute minute-long dexterous movements with minimal working memory, and addresses how counterfactual reasoning underpins all decision-making regardless of the mechanism. He touches on why video prediction has not scaled to world-understanding the way language modeling has, why simulation alone fails without grounding in real data, and the role of compositional generalization in achieving robustness across novel task combinations. The discussion also ventures into labor displacement, education as adaptation, and why conceiving robots as diverse tools—not humanoid servants—may unlock faster progress.
Levine argues that general-purpose autonomous robots are close (median ~5 years to running a household) because modern foundation models supply the prior knowledge and generalizable perception that were missing, enabling a self-improving data flywheel from human-plus-robot deployment rather than waiting for a finished lab product.
- Pretrained VLMs give robots common sense and prior knowledge that did not exist five years ago, the key enabler over past attempts
- A human-in-the-loop flywheel lets robots learn on the job from actions, language, and feedback, so capability scope grows gradually like coding assistants
- Manipulation allows safe mistake-and-correct learning, making it easier to start with limited scope than self-driving cars
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Once a robot reaches a sufficient competence level, it can learn not only from low-level teleoperated actions but also from being instructed through language, so a person standing by and telling it the next steps in words already provides information the robot uses to improve.
“just standing there and telling the robot, "Okay, now pick up the cup, put the cup in the sink, put the dish in the sink," just with words already, actually gives the robot information that it can use to get better.”
Optimal decision making at its core, regardless of method, requires considering counterfactuals—asking whether doing one thing instead of another would be better—whether answered via a learned simulator, a value function, or a reward model; thus the key is not great simulation but a mechanism for answering counterfactuals.
“optimal decision making at its core, regardless of how you do it, requires considering counterfactuals... As long as you have some mechanism for considering counterfactuals and figuring out which counterfactual is better, you've got it.”
The 'good news' for robotics is that a robot trying to accomplish a task has a built-in focusing mechanism that structures perception toward what matters, mirroring how humans exhibit tunnel vision and ignore goal-irrelevant information, which makes embodied models better at absorbing auxiliary data sources than passive video prediction.
“When you have a robot, that robot is trying to do a job. It has a purpose, and its perception is in service to fulfilling that purpose. That is a really great focusing factor.”
The model can execute minute-long dexterous tasks with only about one second of context because the memory demand is an instance of Moravec's paradox: well-rehearsed physical skills are baked into the network and require little working memory, unlike cognitively demanding tasks that require holding many pieces in mind.
“I think this memory stuff is actually Moravec's paradox in disguise.”
Self-generated synthetic experience lets a system rehearse and consider counterfactuals but cannot teach it more about the world; world information must be injected from somewhere, and the most powerful way to generate synthetic experience is from a really good model that itself learned by experiencing the world.
“the synthetic experience that you create yourself doesn't allow you to learn more about the world. It allows you to rehearse things, it allows you to consider counterfactuals. But somehow information about the world needs to get injected into the system.”
The thinking needed per control time step may be surprisingly low because, like animals, planning happens in advance (neural correlates of movement appear before the movement) setting initial conditions that then unroll, with reactive feedback occurring at a more basic level of abstraction rather than re-planning each step.
“If you record from a monkey brain, you will find neural correlates of planning. There is something that happens in advance of a movement.”
Because a robot's sensory stream is extremely temporally correlated, the marginal information from each new observation is far less than the full observation, so past observations and their changes can be represented much more concisely than storing images independently, easing the context/compute burden.
“The marginal information gained from each additional observation is not the same as the entirety of that observation. The image that I'm seeing now is very correlated to the image I saw before.”
Robot arm costs have dropped dramatically—from a $400,000 PR2 research robot in 2014, to $30,000 arms at Berkeley, to about $3,000 per arm at Physical Intelligence now, with belief they can be made for a small fraction of that—driven by economies of scale, technological improvement, and smarter AI lowering hardware precision requirements.
“The robots that we are using now at Physical Intelligence, each arm costs about $3,000. We think they can be made for a small fraction of that.”
Robots should be conceived not as mechanical people but as heterogeneous tools like cars or bulldozers—any size, low maintenance, placeable anywhere—so general intelligence powering diverse robotic forms can outperform humanoid designs and enable solutions like data centers in remote locations.
“We have a tendency sometimes to think about robots as mechanical people, but that's not the case... The better analogy for the robot, it's like your car or a bulldozer.”
Naively scaling data collection 100x does not directly yield the capabilities needed; horizontal scope (more tasks) scales easily, but robustness, speed, and edge-case handling require identifying the correct axes of data, settings, and methods to scale, which is not yet fully understood.
“The challenge here is understanding which axes of scale contribute to which axes of capability.”
The π0 model is a vision-language model adapted for motor control: it reads robot sensory input, does internal processing including chain-of-thought intermediate steps, and routes to an action expert (a separate module because actions are continuous and high-frequency) within an end-to-end transformer structured as a mixture-of-experts.
“Our models, they have a vision encoder, but they also have an action expert, an action decoder essentially. It has a little visual cortex and notionally a little motor cortex.”
Society should plan for eventual full automation of human labor (since humans only have body and mind, with no secret third thing) in a much wealthier economy with some redistribution, but should focus on the unpredictable journey as much as the destination, with education being the single best lever because it provides flexibility against change.
“If there is one single lever that we can pull collectively as a society, it's more education.”
A human-in-the-loop setup where robot plus human work together is much better than either alone and is easier to bootstrap, because the human can label, give hints, and instruct, while the robot gains more opportunity to learn new skills on the job.
“robot plus human is much better than just human or just robot. That just makes total sense. It also makes it much easier to get all the technology bootstrapped.”
Although RL is in principle better than imitation learning, current models use imitation/supervised learning first because effective learning from one's own experience requires already knowing something about the task; supervised learning builds the prior-knowledge foundation that lets the model later figure things out quickly via RL, mirroring how LLMs go from next-token prediction to RL.
“In order to effectively learn from your own experience, it turns out that it's really, really important to already know something about what you're doing.”
Simulation works for human pilots because they are intensely goal-directed toward flying the real airplane, whereas models trained on multi-domain data don't know they must solve a particular downstream task; the key to leveraging simulation is therefore building a foundation model good enough to have emergent ability to use auxiliary data, which itself requires the right objective grounded in real data.
“when a pilot is using a simulator to learn to fly an airplane, they're extremely goal-directed. Their goal in life is not to learn to use a simulator. Their goal in life is to learn to fly the airplane.”
Unlike driving, where mistakes carry severe consequences and are hard to learn from, many manipulation tasks allow a robot to make a mistake, correct it (thereby still achieving the task), and gain knowledge to avoid it in future, making it easier to start the learning flywheel with limited scope.
“With driving, because of the dynamics of how it's set up, it's very hard to make a mistake, correct it and then learn from it because the mistakes themselves have significant ramifications.”
The single biggest benefit recent AI innovations give to robotics is the ability to leverage prior knowledge: pretrained LLMs and VLMs supply abstracted knowledge about the world (identifying objects, locating things in images) from many sources, which is more important than the fact that the underlying model architecture is shared.
“if I had to summarize in one sentence, the big benefit that recent innovations in AI give to robotics is the ability to leverage prior knowledge.”
Traditional factory robots require highly repeatable, precise, robust motions, but smarter AI using cheap visual feedback removes much of that requirement, making robots more affordable because the intelligence can compensate for less precise hardware.
“Traditional robots in factories need to make motions that are highly repeatable. Therefore it requires a degree of precision and robustness that you don't need if you can use cheap visual feedback.”
Unlike text which is discrete tokens, the robot's actions are represented using flow matching and diffusion because they are continuous and require very precise control for dexterous manipulation.
“It actually uses flow matching and diffusion because they're continuous and you need to be very precise with your actions for dexterous control.”
The end goal of robotics is not single tasks like folding a T-shirt but giving a robot a high-level prompt about how you want your household run, after which it autonomously executes for months, learning continuously, applying common sense, handling edge cases, and gathering more information as needed.
“That's the prompt. Then the robot should go and do this for six months, a year. That's the duration of the task.”
Emergent capabilities in LLMs arise not merely from internet data abundance but from compositional generalization: once generalization reaches a level, the model can combine known elements in novel ways (e.g. writing a recipe in the International Phonetic Alphabet despite never seeing that combination), and the same compositionality emerges in robots at scale.
“You can ask an LLM to write you a recipe for making some meal in International Phonetic Alphabet, and it will do it... That's compositional generalization.”
Common sense, the ability to make reasonable inferences about likely outcomes without having to experience the mistake first, is now achievable via LLMs and VLMs (e.g. predicting that a slippery floor sign means caution), a capability that did not exist five years ago and is critical for safe robotic behavior.
“now we can use LLMs and VLMs and ask them questions and they will make reasonable guesses... No autonomous car in 2009 would have been able to answer that question.”
Predicting video has not produced deep world-understanding the way predicting text has, because a visual scene contains unbounded detail (clouds, water molecules, pedestrians) with no inherent prioritization, whereas text is already abstracted into the bits humans care about, providing good representations focused on what matters.
“with text, it's already been abstracted into those bits that we as humans care about. The representations are already there. They're not just good representations, they focus on what really matters.”
Robotics will progress faster than self-driving cars did, not because manipulation is easier, but because 2025 has vastly better generalizable and robust perception and world-understanding technology than existed in 2009, where engineered perception systems could nail a demo but hit a wall on generalization.
“That's not an argument about robotics being easier than autonomous driving. It's just an argument for 2025 being a better year than 2009.”
Levine's median estimate is that within roughly five years (single-digit, not double-digit), robots will be able to autonomously run a household at the level of a human housekeeper, though it will arrive gradually as capability scope expands rather than at one discrete moment.
“I think five is a good median.”
Physical Intelligence aims to build robotic foundation models: general-purpose models that could in principle control any robot to perform any task, which they treat as fundamental to AI because a truly general robot would encompass a large chunk of what people can do.
“Physical Intelligence aims to build robotic foundation models. That basically means general-purpose models that could in principle control any robot to perform any task.”
Robots face a trilemma where inference speed, context length, and model size all consume more inference compute and oppose each other, with current systems at roughly 100ms inference, one second of context, and a couple billion parameters, each orders of magnitude below human-brain equivalents.
“I think currently you have 100 millisecond inference speeds. You have a second-long context and then the model is a couple billion parameters?”
Robotics manufacturing scaling has a self-reinforcing advantage because robots help with physical work, and producing robots is itself physical work, so good robotics should help build more robots—a circular bootstrap unavailable to digital devices, where building computers and phones is work that the computers and phones themselves do not help perform.
“If producing robots is itself physical work, then getting really good at robotics should help with that.”
Dwarkesh questions whether education is the right lever since by Moravec's paradox the things humans benefit most from learning may be easiest to automate (textbooks digestible by AI in an afternoon), to which Levine responds that education's value is flexibility and the ability to acquire new skills and understanding, not the specific facts learned.
“What education gives you is flexibility. It's less about the particular facts you know, as it is about your ability to acquire skills, acquire understanding.”
Getting AI right is not the only priority; a balanced robotics ecosystem requires investing in both software and hardware innovation and infrastructure (compute), and society tends to lose track of these other components when excited about AI progress, so more holistic long-term planning is needed.
“something really important to get right here is a balanced robotics ecosystem. AI is tremendously exciting, but we should also recognize that getting AI right is not the only thing that we need to do.”
Extrapolating AI capex to 2030 yields estimates of 100-300 gigawatts of deployed capacity, implying marginal capex of $2-4 trillion per year, corresponding to physical data centers, chip foundries, and solar panel factories that must be built, raising the question of whether human or robot labor will be the bottleneck.
“many people have estimates in the hundreds of gigawatts – 100, 200, 300 gigawatts... The marginal capex per year is in the trillions of dollars. It's $2-4 trillion dollars a year.”
Despite seemingly AGI-like capabilities, AI companies' cumulative revenues are on the order of $20-30 billion per year, far below the $30-40 trillion value of all knowledge work, making the relationship between LLM capability and revenue mysterious.
“the revenues from these AI companies are cumulatively on the order of $20-30 billion per year and that's much less than all knowledge work, which is $30-40 trillion.”
Rather than one ultimate humanoid robot, the goal should be to find the minimal hardware package that delivers good functionality—reliability and minimal fingers being open question marks, super-precision being unnecessary since feedback compensates—after which capable AI plugged into any robot lets many players innovate hardware optimized per niche, like smartphones converging on touchscreens plus optional extras.
“My job, as I see it right now, is to figure out what's the minimal package we can get away with. I really think about robots in terms of minimal package because I don't think that we will have the one ultimate robot”
The key to overcoming context limits is representing context in the right form for the goal, varying by modality, since humans hold some context symbolically (a shopping checklist as words) and other context spatially or visually (navigating a route), and multimodality has far more potential than image-plus-text alone.
“Representing your context in the right form, that captures what you really need to achieve your goal—and otherwise discards all the unnecessary stuff—I think that's a really important thing.”
The useful way to think about robotics timelines is not the date when development is finished but the date when the self-improvement flywheel starts, because once a robot reaches basic competence it can be deployed, collect real-world experience, and leverage that to keep improving.
“what I tend to think about in terms of timelines is not the date when it will be done, but the date when the flywheel starts basically.”
Levine hopes the best future model will be unified across knowledge work and robotics, and optimistically that the robotics element will make all other capabilities better, because goal-directed embodiment focuses perception and because deep physical-world understanding (beyond language) helps solve abstract problems, as humans use physical metaphors to reason about abstractions.
“I hope it's actually the other way around, that the robotics element of the equation will make all the other stuff better.”
The brain's efficiency advantage over GPUs likely stems from being extremely parallel—doing perception, proprioception, and planning simultaneously—and since transformers are fundamentally parallelizable (made sequential only by position embeddings), an embodied system could run many attentional processes in parallel at different rates rather than one token at a time.
“The brain is extremely parallel. It has to be just because of the biophysics, but it's even more parallel than your GPU.”
Robotics may have an advantage over LLMs for self-improvement because physical tasks produce frequent, observable mistakes that can be corrected and learned from, whereas a wrong answer from an AI assistant may go unnoticed and cannot be easily revisited.
“if you're folding the T-shirt and you messed up a little bit, it's pretty obvious. You can reflect on that, figure out what happened, and do it better next time.”
The main hardware concerns are reliability and cost, and cost matters chiefly because it determines the number of robots, which determines the amount of data available—so lower cost yields more robots and more data, which is the ML team's priority.
“It's just that cost translates to the number of robots, which translates to the amount of data. Being an ML person, I really like having lots of data.”
Making robotic foundation models work requires more than fundamental research; it requires an industrial-scale building effort, more like the Apollo program than a science experiment, with a singular focus on the foundation model for its own sake rather than as a vehicle to publish papers or run a research lab.
“It's more like the Apollo program than it is a science experiment.”
Current laundry-folding policies already exhibit emergent behaviors that were not deliberately trained, such as a robot that accidentally grabs two T-shirts setting one aside, or standing a tipped-over shopping bag back upright, demonstrating compositional capability emerging from learning at scale.
“The robot accidentally picked up two T-shirts out of the bin instead of one. It starts folding the first one, the other one gets in the way, picks up the other one, throws it back in the bin. We didn't know it would do that.”
Compared to multimodal training datasets, the robotic experience data Physical Intelligence has collected is between one and two orders of magnitude smaller, though comparison is hard because robotic time steps are highly correlated and have low information density despite huge raw byte size.
“Maybe a better comparison is to the datasets that are used for multimodal training. And there, I believe last time we did that count, it was between one and two orders of magnitude.”
An economy that gets ahead through a highly educated, high-productivity workforce benefits greatly from automation, because automation multiplies each person's productivity just as LLM coding tools amplify a software engineer's output and robots will amplify the output of everyone doing work.
“Automation is what multiplies the amount of productivity that each person has... Robots will amplify the productivity of basically everybody that is doing work.”
Robotics deployment will likely split between low-cost systems with off-board (cloud) inference that get smarter with good connectivity, and costlier reliable systems with onboard inference for settings like outdoor robots where connectivity cannot be assumed.
“we'll see low-cost systems with off-board inference and more reliable systems. For example, in settings where you have an outdoor robot or something where you can't rely on connectivity, those will be costlier and have onboard inference.”
Dwarkesh introduces the topic and asks Sergey to summarize where Physical Intelligence is.
“Let's talk about robotics. Before I pepper you with questions”