What this covers

David Mindell of MIT argues that decades of experience with semi-autonomous systems in deep-sea exploration, space travel, military operations, and commercial aviation reveal a persistent pattern: full autonomy does not exist. Rather than progressing toward fully self-governing machines, real-world deployments have consistently required human operators in control loops, making them more capable and robust than any attempt at pure automation. Mindell traces this arc across multiple domains, showing how systems designed for complete independence in the laboratory acquired human interventions once moved into complex, uncontrolled environments where the stakes were highest.

The conversation covers how graceful human-automation handoff—not full automation—represents the highest form of technological achievement. The exploration spans the Apollo program's sophisticated fly-by-wire computers, which created rich modes of operator control rather than removing human judgment; the labor-intensive operation of Predator and Reaper drones, which require hundreds of personnel despite their "unmanned" label; and the fact that only 10 percent of commercial airline flights execute exactly according to plan, with human pilots constantly adapting to unanticipated circumstances. Mindell emphasizes that every automated system carries what he calls a human wrapper of autonomy—instructions fed in and data fed back—and embeds the embedded designer worldview of its programmers. He distinguishes between fully automated flight itself, a solved problem, and situated autonomy: the far harder challenge of deploying autonomous systems in environments where other people are present and safety cannot be calculated in advance. The discussion engages driverless cars, the failures revealed by the Air France 447 crash, and why Elon Musk's vision of complete passenger autonomy remains unrealistic.

Sharpest takeaway

Mindell argues that full autonomy is a myth: decades of experience with semi-autonomous robots in deep sea, space, air, and war show that the highest form of technology is not full automation but automation gracefully linked to human operators who can intervene.

  • Every autonomous system carries a 'human wrapper' of instructions and feedback, and embeds the worldview of its designers
  • Across extreme environments, attempts at full autonomy gradually acquired human interventions when moved from lab to field
  • Only 10% of commercial flights go exactly to plan; humans constantly prevent accidents by handling unanticipated situations

The argument · threads7 threads · 28 claims
0.72

Complete autonomy without human involvement is impossible because control is always displaced through code, design, instructions, and feedback loops.

4 pointscentrality 5/5
  • Even autonomous underwater and Mars vehicles are never left fully on their own—you always want to stay in touch, even a couple of bits at a time, and have them report back and bring their data home; one of the book's central themes is that autonomy can be very useful but is always constrained and wrapped in a human wrapper of sending instructions and receiving feedback or data.

    “it's always constrained and it's always wrapped up in a human wrapper of sending instructions and receiving feedback or data.”

  • Full autonomy—a system with no human involvement—is a myth and an asymptote, because we have never built a system without human involvement; the involvement is merely displaced in space or time through coders embedding their worldview, designers, and the wrapper of instructions out and data back, without which the system isn't useful.

    “I said full autonomy is a myth, where there's no human involvement. Because we have yet to build a system that has no human involvement. There's just human involvement displaced in space or displaced in time.”

  • Any automated system is programmed by people, and those programs embody the programmers' assumptions, worldviews, and models of who they think their users are; the claim that the person who thought the problem through years in advance from a cubicle imagined every possible scenario is simply false, because unanticipated things very often happen in the moment—not least the involvement of other people—that humans are good at handling.

    “I said full autonomy is a myth, where there's no human involvement. Because we have yet to build a system that has no human involvement. There's just human involvement displaced in space or displaced in time.”

  • Rather than worrying about machines becoming sentient and pursuing their own interests, one should worry more about machines pursuing the interests of the people who design them, however smart they become; you are much more likely to be killed by a poorly designed robot than by an evil-thinking one, because every machine remains heavily influenced by its designers.

    “I worry more about them pursuing the interests of the people who design them. However smart they may be... I think you are much more likely to get killed by a poorly designed robot than by an evil-thinking robot.”

0.72

Graceful human-machine handoff where operators retain control and can call for more or less automation as situations demand is superior to full automation.

4 pointscentrality 5/5
  • The lunar module had 16 thrusters that no human could command in exactly the right way, making it physically impossible to fly purely manually; even when Armstrong 'flew it by hand' he was flying through the computer, which was beautifully programmed to help the astronauts even while their hands were on the stick.

    “The lunar module is physically impossible to fly it in a purely manual way. It had 16 thrusters and no human could command all 16 of those things in exactly the right way. So you had to fly it through the computer.”

  • Counterintuitively, the Russian spacecraft of the Apollo era were highly automated because they had less sophisticated analog computers, while the more sophisticated digital Apollo computers were used to create a rich human-in-the-loop mode where astronauts could selectively turn off automatic targeting while keeping other digital aids—showing that higher technology does not mean more automation.

    “Actually, the Russian spacecrafts of the time were very highly automated, because they had less sophisticated analog computers. The more sophisticated digital Apollo computers were actually used to create this very rich way of working”

  • On Apollo, the most sophisticated digital fly-by-wire computer did not produce full automation; rather it created a rich way of working where astronauts could turn off automatic targeting yet keep other digital aids—leading to the conclusion that the highest form of technology is not full automation or autonomy, but automation gracefully linked to the human operator who can call for more or less automation as the situation demands.

    “Actually, the Russian spacecrafts of the time were very highly automated, because they had less sophisticated analog computers. The more sophisticated digital Apollo computers were actually used to create this very rich way of working”

  • Highly expert operators who land lunar modules, space shuttles, and auto-land airliners turn off automation not from ego but because being more in touch with what the machine is doing gives them a better chance of responding to a failure or anomaly at the last moment; the best person to handle unforeseen surprises is not the programmer working years before but the person physically in the environment whose rear end is on the line.

    “these operators, who are very highly expert, really believe that if they are more in touch with what the machine is doing, they have a better chance of responding to something if there is a failure or some anomaly at the last moment.”

0.68

The hard unsolved problem is situated autonomy—applying systems in human environments where other agents are present, unpredictable, and require social understanding.

3 pointscentrality 5/5
  • Fully automated flight—takeoff, fly through weather even with engine failure, and land at another airport—was solved 20 years ago; the hard, barely-scratched problem is situated autonomy: applying autonomous systems in a world where other people are present, flying over heads of people at risk, sharing airspace, which is the real problem for drones, the FAA, and driverless cars, and is not amenable to a purely analytical solution.

    “if you want to talk about a fully automated aircraft--take off from an airport, fly even through weather, even with an engine failure, and land at another airport--we solved that problem 20 years ago. That's a solved problem. But to try to do that where you take off from an airport where other people are using... that's a problem we're only barely scratching the surface on.”

  • The frontier and greatest potential of robotics lies in situating autonomous systems within human environments—which includes almost anything economically valuable and productive—by taking systems engineered for full autonomy and bringing them into the human world to respond and react to human systems and behaviors in new, situated ways; the elements are present for great progress though we are not there yet.

    “the frontier is situating robots and autonomous systems within the human environments. And when you say 'human environments' you include almost anything that's economically valuable and economically productive.”

  • Computer science shows little progress at understanding social relationships—how to observe people going in and out of a building and extract their intentions, plans, and whether behaviors are normal or abnormal—because that depends on richly understood human context; much bridging remains between the AI-robotics world and people who deeply understand human behavior, and the rhetoric around full autonomy reveals that the technical community's grasp of the social world is still rather primitive.

    “I don't see a whole lot of progress in the computer science world at really understanding social relationships... the technical community's understanding of the social world is still rather primitive.”

0.64

Autonomous systems are brittle, narrow, context-dependent solutions that fail when moved outside conditions they were engineered for.

2 pointscentrality 5/5
  • Most autonomous systems, when they succeed brilliantly, do so within a particular, well-thought-through, narrow set of conditions and are very difficult to move outside the context for which they were created; they are still fairly fragile, brittle solutions, which is why an engineer who builds them finds them frustratingly dumb rather than threatening.

    “most autonomous systems that we make, when they succeed brilliantly, they succeed brilliantly out of particular, well-thought-through, kind of narrow set of things. And they are very difficult for them to move outside of the context for which we've created them.”

  • Driverless cars work well only when bandwidth is good, near cell towers, sensors are clean and working, and the area is perfectly mapped; but driving away from cell links, on unmapped dirt roads, in snow that changes the 3-D terrain by nine feet overnight (as in a Boston winter), means users must move in and out of autonomous modes—reintroducing the Air France 447 handoff problem, since the world is not fully wired for perfect, everywhere, all-the-time autonomy.

    “there's no question that when you have good bandwidth and you are near a cell tower and the sensors are all working at their highest order... you ought to be able to have access to great features. But you really are going to have to be able to move in and out of those features”

0.58

Driverless cars and fully autonomous aircraft are unrealistic near-term goals; partial automation improving safety is good but full hands-off control is not imminent.

4 pointscentrality 4/5
  • Elon Musk and Google's vision of a fully driverless passenger car in which you sleep in the trunk while your kids are strapped in the back is not realistic and is a long way off, because there are numerous ways human interventions will be necessary; automated features that improve safety are good and coming, but full hands-off autonomy is not near.

    “to have a car that you drive down the highway at 80 miles an hour and sleep in the trunk while your kids are strapped in the back seat--I think we're a long way from that.”

  • Unlike Google's stated push toward complete autonomy, almost all car companies are taking a more cautious approach—BMW's leadership says people buy their cars because they like to drive them, so removing that would be crazy—and these companies, more familiar with engineering life-critical road systems, will compete in the marketplace while it remains hard to imagine regulators allowing a car without a big red Stop button, which itself begins a handoff that must be engineered.

    “to have a car that you drive down the highway at 80 miles an hour and sleep in the trunk while your kids are strapped in the back seat--I think we're a long way from that.”

  • There is no evidence yet that driverless cars save lives; while a high proportion of car accident deaths are caused by human error, we know far less about how people drive under normal circumstances, where they smooth out rough edges—knocked-over stop signs, broken traffic lights—and muddle through, just as only 10% of flights (and likely car trips) go exactly to plan.

    “there's no evidence that we're going to save lives yet. There may well be. But again, we know a lot about accidents... But what we know a lot less about is how people drive under normal circumstances. And people are extremely good at sort of smoothing out the rough edges”

  • The prevailing mythology that we are moving in sequence from human to remote to autonomous is wrong; instead the three modes are all converging, so the future driving system will be a mix of human and remote autonomous systems—we will see autonomous features and autonomous driving for certain times, places, and applications, but never a uniformly hands-off world.

    “to have a car that you drive down the highway at 80 miles an hour and sleep in the trunk while your kids are strapped in the back seat--I think we're a long way from that.”

0.58

Human pilots and operators prevent most accidents by constantly correcting small errors and adapting to unforeseen circumstances during normal operations.

4 pointscentrality 4/5
  • The Air France 447 crash was a story of a failed handoff: automation found a relatively minor fault, handed control back to pilots too suddenly and ungracefully, and because the pilots had lost flying skills from over-reliance on automation, they lost control of what was actually a perfectly good airplane, illustrating what goes wrong when human-machine control exchange is not properly engineered in life-critical extreme environments.

    “the Air France story is a story about a failed handoff, where the automation onboard an airplane found a relatively minor fault and handed control of the plane back to the human pilots, too suddenly and ungracefully surprised them. And they had lost some of their skills flying too much with the automated systems and lost control of the airplane.”

  • By and large computerized airliners are very, very safe and have played a role in the tremendous decrease in commercial airline accidents over the decades, even though the computers are not perfect and people constantly correct small mistakes.

    “By and large, computerized airliners are very, very safe; they certainly have had a role in the tremendous decrease in accidents in commercial airline flight over the decades.”

  • The FAA's study of cockpit automation estimated that only 10% of commercial airline flights go exactly according to plan; in the other 90%, there is always some change in routing or circumstance that human pilots adapt to, and without people in those loops there would be many more accidents.

    “The FAA's (Federal Aviation Administration's) study of cockpit automation estimated that the number of commercial airline flights that go exactly according to plan is 10%. And in the rest of the 90% case there's always some change”

  • Because accidents get disproportionate attention, study, and resources while normal operations are poorly understood, one must be careful not to evaluate automated systems only through accidents; in normal situations people are constantly preventing accidents by correcting small errors and responding to changing situations, and without people in those loops there would be many more accidents.

    “The FAA's (Federal Aviation Administration's) study of cockpit automation estimated that the number of commercial airline flights that go exactly according to plan is 10%. And in the rest of the 90% case there's always some change”

0.57

Autonomous system designers must give users transparent input into value-laden tradeoffs embedded in algorithms so machines reflect user priorities, not hidden programmer choices.

7 pointscentrality 3/5
  • The Predator and Reaper drones, designed according to a dream of full autonomy at great expense, ended up embedded in the human world like every other system; far from being inhuman killing machines, they take hundreds of people to operate—so labor-intensive that the Air Force banned the term 'unmanned' for them—and their operators spend most time watching houses and interpreting fuzzy video for human intent, a task people still do better than machines because it requires human context.

    “they take hundreds of people to actually operate them. And it's actually a problem because they are so labor-intensive to operate them... the last thing they are is inhuman killing machines.”

  • Contrary to expectations that deep-sea robotic vehicles would be cheaper, safer, and replace humans moving toward full autonomy, they were neither cheaper nor safer than manned vehicles but instead fundamentally changed the nature of scientific work—turning solitary exploration into a shared, social, group experience via real-time fiber-optic relay to control rooms of dozens, and ultimately pushing humans to a different place rather than replacing them.

    “one thing that surprised us right away was that the robotic vehicles weren't cheaper and they weren't safer than the manned vehicles... What it did do is push the people to a different place, and changed the nature of the work that they did.”

  • Every time you drive you make tradeoffs—e.g., turning the risk knob up when late or down when carrying kids—and so does every autonomy algorithm, which works by optimizing cost functions balancing factors like fuel efficiency versus performance that are often in conflict; therefore systems should be designed so the user has transparent input into those value-laden decisions, so the car drives according to your values and priorities rather than a programmer's hidden choices.

    “So does practically every autonomy algorithm. They work by optimizing cost functions: What is the balance between fuel efficiency and performance on this particular trip? And very often those values are in conflict”

  • Many scientists resisted remote robotic exploration in the 1990s, feeling it wasn't really science unless one physically inhabited the place studied; but since even in a manned submersible you are encased in a hull protecting you from the elements—itself a kind of remote presence—it took a long time for people to accept that using robots for remote exploration is also genuine exploration.

    “A lot of scientists felt that remote science wasn't really science... So, it's a remote presence of its own kind. But it took a long time for people to accept that using robots to do remote exploration might actually also be exploration.”

  • A young person who wants to be part of the robotics evolution should study computer science and the social sciences at the same time, because designing engineering systems that are situated within social and political systems makes for better, more successful systems—a rejection of 'scientism,' the idea that you can calculate everything about the world in advance and that the user is an idiot with nothing to add.

    “Study computer science and the social sciences at the same time.”

  • The Air France computer abandoned control because its programmers told it that it could not fly without perfect data from the pitot tubes, even though aircraft can fly without that data; the binary rule of 'if all data isn't perfect, check out altogether' seems reasonable but ignores the dangerous distress scenario it hands to human pilots without warning.

    “the human programmers had said, 'If all the data coming in isn't perfect, then you've got to check out altogether.' And that's basically what the computer did.”

  • The precision-guided 'smart bombs' of the 1991 Gulf War, with all their computers and lasers, are actually the dumb bombs because they go only where we want; the genuinely 'smart' bombs are the scary ones whose destination you don't know due to wind or system failure—autonomy, like a bullet's flight after the barrel, represents the failure of the system when it deviates from human intent.

    “those are not the smart bombs. Those are the dumb bombs. Those are the ones that are going only where we want them to. The smart bombs are the really scary ones where you drop it out of the plane and you don't know where it's going to go”