
World's Top Researcher on AI, LLMs, and Robot Intelligence
What this covers
This is a conversation between Patrick O'Shaughnessy and Sergey Levine, co-founder of Physical Intelligence, that starts with the history of robot learning. Levine notes that end-to-end learning for driving goes back to a tiny neural network in the mid-to-late 1980s. He describes a robotics field that long resisted learning because physics seemed to require explicit programming, and says end-to-end learning and the Bitter Lesson are still contested. He also covers the collapse in hardware costs, from a roughly $400,000 PR2 to arms costing a fraction of his $30,000 lab robot. Moravec's Paradox explains why engineers keep underestimating the difficulty.
The middle of the discussion ranges across technical fault lines. Levine calls the fusion of generative AI's inherited human knowledge with deep reinforcement learning's capacity for discovery an unsolved problem. He explains how chain of thought lets a robot reason through a sub-task before moving, and how multimodal models supply common sense but are poorly grounded physically. He contrasts humanoid acrobatics, built on simulation, with manipulation, built on real-world data, and leaves open whether the two will merge. He also takes up the monkey-tool studies as a view of embodiment, and argues that robotics will end up with a toolkit of body types rather than one dominant humanoid.
The more contentious turns are his strongest counterargument to the bitter lesson, that known physics should be plugged in rather than relearned, and his admission that nobody knows how much data is enough. He compares robot deployment to coding tools, where humans and machines co-evolve instead of one replacing the other. He also reflects on when a researcher should persist with an idea, and on a recent shift toward supervising robots with high-level semantic labels, so that a person can coach one by talking to it.
Levine argues that robotics will be unlocked by general-purpose robotic foundation models trained on diverse data, because generality with respect to improvement, not hand-engineering for one body or one demo, is what scales, and web-pretrained common sense combined with reinforcement learning is the path to handling open-world situations.
- Training at full generality is easier in the long run than special-casing narrow applications, as happened with language models
- Systems that improve from their own experience and from high-level supervision generalize better than hand-designed or narrowly trained controllers
- Multimodal LLM knowledge supplies common sense for unusual situations, but must be grounded in physical control
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The most delicate decision in research is when to keep pushing on an approach versus switching to explore new ones; quitting too early can miss a breakthrough just short of the answer, while persisting too long can waste years, and people with an instinct for this timing have a large advantage.
“when do you try new things or is when do you stick with what you're already trying? And that's very very delicate”
Scientific progress in machine learning comes from many people trying many things, and failures and bad ideas are instructive; robot learning might not have advanced as it did without the controversy over end-to-end learning, so one should not judge researchers by a list of home runs alone.
“these advances happen because lots of people are trying lots of things, and even some of the failures are actually very instructive”
End-to-end learned driving is decades old: an Alvinn-style system in the mid-to-late 1980s (Alvin, circa 1986 or 1987) drove highways using a tiny neural network that took camera input.
“Alvin was I think it's 1986 or 87 and that was a driving system that was demonstrated to drive on highways controlled by a neural network and then from a camera”
Studies of monkeys using tools show that neurons representing the hand shift to represent the tool tip, so a tool becomes a real physiological extension of the body.
“they activate based on the location of the tool tip, not based on location of the hand”
Compositional generalization means a model can combine learned building blocks to solve novel problems: a language model can write paragraphs in the International Phonetic Alphabet, a form it never saw in that form, because it understands that the same content composes with a different alphabet, and robots should likewise combine a learned repertoire of skills.
“that is compositional generalization. It means that you have never seen this particular language... but you understand that it's compositional”
Moravec's paradox holds that tasks easy for humans (picking up a cup) are hard for machines and vice versa, because human intuition for physical skills is the product of evolution and therefore feels effortless, creating a cognitive bias in which engineers underestimate robotics difficulty.
“we kind of have a cognitive bias to think that things that are easy for us will be easy for the machine”
Robot learning has been hard because a usable system must be cheap to train per application, handle long-tail scenarios with common sense, and be robust, fast and reliable; naive per-task data collection fails the cost requirement, so general models that need less data per new task are essential.
“that's not cost-effective because then you go on to the next application and you go through that process all over again”
Language models displaced domain-specific NLP systems because they can leverage much broader data, including weakly labeled web data, which builds a foundation of world understanding on which specific applications are far more effective to build.
“you actually learn more about the world. So, you establish like foundation of world understanding and then on top of that foundation turns out to be much more effective to build out different applications”
The bitter lesson says programmers should not force a machine to think the way they think it should, but should let it learn from data; Levine argues that to get generality, especially generality in the machine's ability to improve, learning from data must be primary.
“The bitter lesson says that you should not program the machine to think the way you think it should think, but you should let it learn from data”
Robot hardware has become dramatically cheaper: a PR2 cost about $400,000 when Levine began in robotics about a decade earlier, his Berkeley lab robot cost about $30,000, and each current arm costs roughly a tenth of that, with further reductions expected, and low-cost arms are only useful for general robotics because they lack the precision industrial control requires.
“I worked with a robot called a PR2, which I believe had a cost of about $400,000... each arm on this thing is maybe a tenth of that”
Robotics lacks an internet-scale dataset, but a world-understanding foundation is even more important in robotics, because training one model across many tasks and physical systems lets it grasp physical interaction and makes new skills learnable very rapidly, as humans do.
“this notion of understanding the world, if anything, is actually more important in robotics”
Combining skill practice with prior knowledge is a hard unsolved problem across all of AI, because generative AI reproduces human abilities while deep RL discovers things humans have not thought of (AlphaGo's Move 37), and the field has not yet fused these two strengths.
“generative AI is impressive because it can reproduce some of the things that humans can do... Deep RL is impressive for the opposite reason. It does things that humans hadn't thought of”
Chain-of-thought reasoning gives robots common sense: before moving, the robot verbalizes the sub-task implied by its instruction (for example, that cleaning the kitchen means picking up the plate), and those intermediate inferences draw on web-scale pretraining to handle edge cases.
“the robot enters a scene... instead of directly starting to move, it thinks about what it was asked to do... Pick up the plate. And then it goes and does it”
The most recent change is that multimodal LLMs contain common-sense knowledge that can be pulled into robots to handle unusual situations, but they are poorly grounded in physical situations and must be plugged in carefully, since they lack robot context.
“multimodal language models are really good at pulling in knowledge... They're not very good at like grounding that knowledge in physical situations, but they know stuff”
Building general-purpose robot models at full generality may in the long run be easier than special-casing narrow application domains, just as solving natural language tasks in full generality proved easier than narrowly targeting machine translation or sentiment analysis.
“doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains”
Nobody knows how much robot data is needed for truly generalizable embodied AI, and the key is not to quantify the robot data set but to deploy systems useful enough to go into the world and gather more data themselves, creating a flywheel analogous to Tesla's fleet data.
“the key is to get a system that can go out to the world that's useful enough, that does a wide variety of different things, and that can keep pulling in more data”
Humanoid acrobatics rely heavily on simulation with almost zero real-world data, while robotic manipulation succeeds by using very little simulated data and large amounts of real-world data with large foundation models; these opposed pipelines are unresolved, and a synthesis may eventually emerge.
“There's a particular pipeline that makes that work which is very heavily reliant on simulation and actually very light on real world data”
Robotics has been historically resistant to learning because physical systems seemed to require explicit programming of physics, and the field took a long time to accept that learning systems can figure out physics themselves; learning is now broadly accepted, but end-to-end learning and the bitter lesson remain contested.
“you don't necessarily need a physics simulator inside your robot when it's planning, we can actually have a learning system figure all that stuff out”
The strongest argument against the bitter lesson in robotics is that reliable operation in complicated open-world settings requires using what is already known about the physical world, which is codified in textbooks, so engineers should plug that knowledge in rather than relearn it from data.
“if you want something reliable in a really complicated open world setting, then you can't afford not to use what you already know about the physical world”
Coding tools did not eliminate software engineers; they raised individual productivity while requiring work to make them usable, and robotics will likely follow a similar co-evolution, with some tasks done by robots alone, some by humans with robots, and others where robots make humans more productive, rather than humanoids replacing workers.
“It's not like coding tools came on the scene and suddenly we don't need software engineers anymore. It's that the coding tools increase the productivity”
Across different robot bodies, the fundamentals of how to interact with objects, how things move, and how causality works are conserved, so a foundation model can be adapted via fine-tuning or in-context information to each body.
“the fundamentals of how you interact with objects, how things move in the world, how causality works, like that's all conserved for all these different systems”
Robotics will not converge on one dominant form like the humanoid; it will be more like a toolkit of many body types, and a shared foundation model is the platform that makes that toolkit usable.
“It's going to be more like kind of a toolkit where you can put together all sorts of like really cool applications”
Chain-of-thought supports long-horizon tasks, and about six months before the interview the team found that its models could be improved simply by supervising them with high-level semantic labels on existing experience rather than adding more teleoperated low-level actions, shifting the bottleneck from low-level motor skill to mid-level scene interpretation and step selection, so a person can coach the robot by talking to it.
“what we tried, kind of on a whim, is to see... what if we just add more data labeled with the semantic command... And that actually helps”
The central design principle is generality with respect to improvement: hand-designed controllers require a human engineer to improve them, learning-based perception requires only human labelers, and a system that learns autonomously from its own experience needs no human in the loop, so that is the criterion driving decisions.
“the most important thing to get right is... to get it to be general with respect to how it can be improved”
Robotics demos are optimized for impact by picking one cool task and controlling everything else in a pristine setting, whereas real generalization means doing a mundane task any human could do in any situation, so generalization is systematically underrepresented in exciting demos.
“effective robotic learning, effective generalization isn't actually the optimal way to have like a really exciting demo”
The near-term research focus is middle-level reasoning, because low-level physical behaviors can now be acquired but generalizing them requires common sense, and LLMs' text-to-text representations are not the best internal representation for an embodied system, which may need spatial, semantic, or other representations.
“LLMs make certain kinds of representations very convenient... that's not necessarily the best representation for what an embodied system needs to do”
Progress in form factor innovation has been constrained in traditional robotics because building a new robot requires characterizing dynamics and system identification; a general robotic foundation model would let anyone assemble a robot in a garage and get it moving, which could create a powerful engine of experimentation.
“If you could just put together a robot in your garage, load up a robotic foundation model, and tell it to do a bunch of stuff”
A general robotic foundation model could act like the personal computer, letting many people hack together new robotic applications and triggering a Cambrian explosion of applications, whereas today building any new robot application requires a monstrous stack that solves the entire intelligence problem.
“something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics application... you need to basically solve the intelligence problem”
Naively showing a multimodal LLM a picture and asking what it would do fails, because the model does not know it is a robot, what the robot looks like, or what is going on, so connecting LLM knowledge to robot control is a real technical challenge.
“You can't just like show it a picture and say, What would you do here? because it doesn't have the context”
Robots can exceed human speed on tasks like plugging in cables: humans pause to process what they are doing, and teleoperation is even slower; removing those pauses, and speeding up with demonstration-based success criteria and RL, makes robots much faster, and the bottleneck is processing speed, which computer science understands well.
“when a person does this without having practiced a lot, you pause frequently... speeding up processing is something that people understand quite well in computer science”
The under-asked question is how someone should start using autonomous robots for a task, including whether to set up teleoperation, modify the task, or design new hardware, and people often assume that any easily collected data (such as videos of humans) is the right data, which is often wrong and domain-dependent.
“machine learning requires data, so let me just like figure out something that will collect data. That's not often the best assumption because you need the right kind of data”
Hardware-software cost reduction is a constellation of technologies: low-cost arms would be useless in traditional industrial settings requiring precise control, but learning-based control makes them practical for general-purpose robotics.
“the kind of um uh low-cost arms that we have here, they wouldn't be useful in an industrial setting because traditional control methods that rely on a great deal of precision wouldn't be able to use them.”
In the 'Robot Olympics' test of everyday tasks proposed by Benji Holson, the team's general system solved almost all of the tasks (opening doors, washing a greasy pan, using a plastic bag to pick up dog waste) without special development, failing only on turning a dress shirt inside out (grippers too big for the sleeve) and peeling an orange with fingers (used a knife tool instead), which showed the power of generality.
“we could solve almost all of them... we didn't like develop anything special for this”
A key uncertainty that changes how companies should prepare is whether robots will rely mainly on teleoperated demonstrations or on reinforcement learning from autonomous experience; a 90/10 mix in one direction versus 10/90 in the other would require dramatically different data strategies, and the answer will emerge over the next few years.
“Will the robots rely more on demonstrations or on reinforcement learning from autonomous data?... Is it 90/10 or 10/10 10/90?”
Levine's early approach of blank-slate practice of a single skill worked in limited settings but failed to generalize to open-world conditions, and parallel learning across roughly 20 robots at Google generalized yet produced savants that cannot handle tail cases.
“the next thing I tried... was to paralyze it across many robots. So collective learning... it becomes this kind of savant of this particular task”
The most significant technical risk is breadth: well-defined but slightly chaotic environments like hotel rooms or restaurant kitchens are controllable, but homes contain unexpected events, so a robot must always do something sensible and acceptable, which is the hardest part of the problem.
“the place where I would see the biggest technical risk is dealing with the breadth of different situations”
Scientific progress and robotics research are accelerated by many people trying many things; the company's engagement strategy of open-sourcing models is intended to let many companies experiment, since there is power in many people trying lots of things.
“that's why physical intelligence has really put a premium on engagement like we've open sourced our models.”
The hardest tasks for robots to achieve are those involving interaction with people, such as elderly care, helping someone out of bed, and caring for small children, which are harder than people appreciate because humans are evolved for these skills and do them without conscious deliberation; this is the pinnacle of Moravec's paradox.
“So I think like elderly care, taking care of small children, I think those things are going to be hard and they're probably going to be harder than people think”
Scaling up robot manufacturing (making 100 million units) is important but not the most urgent problem; the better sequence is to solve the hard intelligence problem first, since making robots at scale is harder if you do not know what software or form factor will run them, and general robot models reduce that uncertainty before scale-up.
“making a robot at scale is even more difficult if you don't know what kind of software is going to run it afterwards”
Demos are valuable when they illustrate challenges on the road to something useful and are used honestly in service of a mission, since they show people what to expect and set a challenge, though a demo unconnected to a useful mission has less value.
“demos that are used correctly in service to a mission can provide people with an illustration of what to expect”
Levine is more optimistic than established robotics researchers but more pessimistic than robotics entrepreneurs; robotics has a long history of few successes, most useful robots still run on tightly controlled programs, and each mountain climbed reveals another, so caution is warranted even as he sees puzzle pieces that could address past failures.
“I'm on the optimistic end when it comes to um established robotics researchers and on the pessimistic end relative to robotics entrepreneurs”
Organizations can foster progress by empowering researchers' pet projects, as OpenAI did with ChatGPT (originally John Schulman's pet experiment rather than a corporate strategy), and this kind of atmosphere is valuable but very hard to create.
“ChatGPT was, you know, basically John Schulman's pet experiment for a while... there's something pretty inspiring about organizations that empower people to have pet projects turn into world-changing successes.”
Early-stage applications of LLMs are accessible, letting anyone prototype cool applications by prompting a model; the speaker argues that rapid iteration by many smart people is the best route to discovering uses that no one could predict.
“there's an amazing kind of a power to having lots of people, lots of smart people rapidly iterating and prototyping lots of things”
The goal of Physical Intelligence is to develop robotic foundation models that can control essentially any embodied system to do any task, analogous to how language models are converging on handling any task expressible in language.
“the goal of physical intelligence is to develop robotic foundation models that can control basically any embodied system to do any task”
The timeline for useful robots is highly uncertain because the field faces a bootstrap problem: robots must cross an activation-energy threshold of usefulness to be deployed and collect open-world data at scale, and the timing depends on whether data comes from teleoperation, autonomy, shared autonomy, or coaching.
“because that's such a like a sudden kind of event with getting past the activation energy, I think there is a lot of uncertainty”
As machine learning matures, domains where data is easy to collect will fall into the easy bucket even if physically intricate, while domains requiring common sense, multi-level abstraction, and linking web knowledge to physical skills will remain hard and need further technical advances.
“domains where collecting data straightforward, they actually end up falling into the easy bucket over time, even if they are physically intricate”
Physical analogies such as 'momentum' in business and Feynman's 'spin' for subatomic particles let people reason effectively about unfamiliar domains, which suggests humans are primed for physical intelligence and that understanding physical causal structure may be something special that LLMs may or may not capture.
“people are remarkably good at using physical analogies to understand other situations”
The most surprising finding has been that dexterous behaviors emerged without special techniques, and that one model generalized across embodiments, including multi-fingered hands and different degrees of freedom, after data collection and fine-tuning but without changing the model or prompting it with the robot's identity.
“we could also get these systems to perform very dexterous behaviors without really doing anything particularly special for that”
The platform uses three cameras (one on each wrist and one on the base) and has no touch or force sensing; the design is deliberately bare-bones and low-cost, and a good learning method can compensate for deficient sensing, since wrist cameras act as touch sensors in disguise by revealing local deformations.
“this platform here has three cameras, one on each wrist, and a base camera. It doesn't have touch sensing, it doesn't have force sensing”
Research papers are a better source than social media demos for understanding the true state of robotics, since demos are meant to show the edge of capability; the most reliable insight often requires talking to the individual researchers to learn the inside story.
“the public-facing artifacts, like the demos and the videos... are often actually not very good for providing a sense for the true underlying state of things”
The base architecture is a vision-language-action model, an LLM adapted for robotic control, trained first on text, then on web images for visual understanding, and finally on diverse robot data.
“the basic foundation is what's called a vision language action model... first trained on text data, then they're adapted with lots of image data from the web... and then they're adapted to robots”
The company's strategy is to make its robots as cool as possible subject to the constraint that they are useful; it sets out to build general capabilities and uses the toughest, often cool-looking challenges as stress tests rather than targeting espresso or laundry directly.
“the strategy we've taken is subject to the constraint that it's useful, make it as cool as possible”
Common sense in robotics means applying semantic inferences from knowledge learned in other domains to the current physical task, and it is the opposite of muscle memory, which is practiced skill executed on autopilot.
“you can think of common sense as sort of the opposite of muscle memory”
If there is still no robot doing dishes in 2050, the most likely explanation is a long tail of human-technology interaction challenges, such as comfort with imperfection: people will accept robots that learn from mistakes in some tasks but not others, like around small children, and deployment timing depends on that.
“there is a long tail of challenges that has to do with the interaction of technology and and people”
In the long run, general learned robot systems could enable very small and very large machines in medicine and surgery, and could remove the need for robots to resemble humans or even be human-controlled, since current robotic surgery is done entirely through teleoperation.
“in medicine, in surgery where we not only might in the long run not be limited to robots that look like humans”
Repeatedly showing something people thought impossible, as Boston Dynamics does, revises expectations of what is possible even when caveats exist, and organizations that empower pet projects, as OpenAI did with ChatGPT, can produce world-changing successes.
“there is a lot of value in repeatedly showing something that people wouldn't have thought possible”
Whether a robot should be deployed under human teleoperation or as a partially autonomous system is domain-dependent; some settings may accept offsite human control, while people may not want a home robot constantly controlled by a remote person.
“some domains where deploying a system under human control makes a lot of sense. There's some domains where deploying a partially autonomous system is very reasonable”
Great researchers do not share a single personality profile; they can be driven by a desire for novelty without caring whether their work is useful, or by a passion to solve a specific problem, and both types can be highly effective.
“The one constant is that there's no constant, basically”
The Roomba is the best-selling consumer robot of all time, which is surprising.
“I think to this point Roomba is like the best-selling robot of all time in in the consumer category”
At Google in 2015, Levine's bottom-up 'arm farm' project put a couple dozen idle robots from an unused warehouse into a lab to collect data, approved by Jeff Dean and Vincent Vanhoucke even though he was a junior research scientist, showing the leverage that organizations can grant individuals.
“one of the projects that I did... in 2015 was a colloquially called as the arm farm”
The company is agnostic about hardware design choices such as the number of cameras, touch sensing, or whether the eventual model includes a language model, and expects to try many options; only the level of generality is fixed.
“I don't know if the correct design for a robot is to have three cameras... I think we're very agnostic to that”
Humanoid robots have real value because they are compelling and help people imagine the future, but a humanoid is only one of many possible robot bodies, and intelligence should be tackled in a general way rather than within one specific body.
“I don't think we should be tackling intelligence in the context of one specific body”
Levine began robotics work in 2014 as a postdoc with Pieter Abbeel at UC Berkeley after a computer graphics background, reasoning that it was a bet on applying AI learning to physical systems; his persistent goal has been AI systems that improve the more they practice.
“I started working in robotics in um 2014, and that was actually after I finished my um graduate degree and I started a post doc with Professor Pieter Abbeel at UC Berkeley.”
Deep reinforcement learning from the early 2010s is likely a historical milestone because it allows systems to exceed human-level performance, which Levine considers essential for robotics; the advent of multimodal LLMs adapted for control is another major advance, though its historical status is uncertain.
“the first deep reinforcement learning systems which were in in the early 2010s, like those are probably a milestone”
After a policy has been practiced, reinforcement learning from its own experience improved an espresso-making robot's robustness, speed and throughput, and Levine says that work is far from finished.
“that system practiced making those espressos many, many times, and used that to improve robustness, improve speed, improve throughput”
The company is spending time on the space of possible product shapes that might bootstrap data collection, but it has no concrete answer; its 2026 plans include experimenting in that space by prototyping real-world applications and observing what goes wrong.
“there's a space of possibilities and a lot of what we actually are planning to do in 2026 is also experiment with different things in that space”
Most advice to companies about adopting LLMs is hard to give because the technology changes so rapidly; the speaker's point is that uncertainty about the technical approach makes preparation hard, so rather than predict the answer, companies should watch which approach wins.
“it's a very good question. It's also a very difficult question because the technology is changing so rapidly”
The kindest things others have done for Levine were bets on his potential: Jeff Dean and Vincent Vanhoucke backing the arm farm, Pieter Abbeel hiring him as a postdoc with zero robotics experience, and an NVIDIA hiring manager giving him an internship as a college sophomore.
“I think the hiring manager for that also took a bet on me”
A kitchen-cleaning robot demo that looks ordinary out of context (picking up plates) is nonetheless frontier-pushing, because the robot was placed in a home it had never had training data from and still succeeded.
“we put it into that home just for that demo and it hadn't never had training data from that setting.”
Levine states that the 'bootstrap' of robot data from useful deployed systems and the 'weakly labeled' web data analogy shows that world understanding is the foundation for application-specific performance, and that Physical Intelligence bets on general over narrow.
“that's kind of a case study of how changes in technology will dramatically alter this”
Boston Dynamics' new Atlas is a favorite because it is both human-like and not human-like, with unusually wide joint range of motion and agility that enable impressive demos.
“I do really like the Boston Dynamics robot the new especially the new version of the Atlas”
The host discloses that he is an investor in Physical Intelligence because he believes it is one of the most important companies tackling robotics.
“I'm an investor in Physical Intelligence because I believe it's one of the most important companies tackling the problem of robotics”