What this covers

Derek Leben joins Sean Carroll to examine how to build ethics into artificial intelligence systems that operate beyond controlled domains. The conversation spans philosophy, game theory, and real-world cases—from self-driving cars to autonomous weapons—where machines must choose courses of action in ambiguous circumstances with moral weight. Leben argues that programming choices are inescapably normative; there is no neutral stance, only a choice between explicit and hidden assumptions. He defends contractarianism as the best moral framework to implement, grounded in John Rawls's theory of justice and the maximin principle: rather than maximizing overall happiness, an AI system should distribute primary goods—health, safety, opportunity, essential resources—to raise the floor for the worst-off.

The discussion moves through concrete problems: why a car cannot treat all collisions as equally bad, how Asimov's prohibitions break down when every available action causes or allows harm, and whether machines trained by imitating human judgment will inherit human bias. Leben stakes his case on game-theoretic grounds—that among internally consistent moral theories, contractarianism uniquely produces cooperative behavior across all scenarios, outperforming utilitarian summation in cases like the utility monster where maximizing total happiness would justify severe injustice. Carroll presses on intuition as a check on theory, leading Leben to explain why he discounts moral intuitions given their historical track record. The pair also examine whether moral facts are objective—Leben frames them as contingent on shared goals, akin to health recommendations—and how the need to instruct machines literally clarifies what human moral philosophy has often left vague.

Sharpest takeaway

Leben argues that designing ethical AI is unavoidable and that the best moral framework to program is a Rawlsian contractarianism—applying the maximin principle over primary goods—because among internally consistent moral theories it best fulfills morality's evolutionary function of producing cooperation among self-interested agents.

  • Any choice in programming a machine reveals normative assumptions, so avoiding ethics is impossible
  • Moral theories should be evaluated by which one best produces cooperative behavior, formalized via game theory (Nash equilibria, Pareto improvements)
  • Contractarianism's maximin over primary goods (health, safety, opportunity) outperforms utilitarian summation in producing cooperation across all scenarios

The argument · threads6 threads · 28 claims
0.85

Autonomous machines in warfare, medicine, and transport must embed contractarian reasoning; top-down articulation is ideal but reliable bottom-up matching suffices.

10 pointscentrality 5/5
  • Asimov's first law (don't cause or allow harm to humans) works in ordinary circumstances but breaks down because it is vague about what counts as harm (property, dignity, insults) and because in moral dilemmas every available action either causes or allows harm, making the rule inapplicable.

    “in some scenarios, you can't avoid either causing or allowing harm to others, and Asimov's law simply breaks down.”

  • For security and military robots, contractarianism via maximin would favor neutralizing a threat over killing the threatening agent, because killing makes that agent maximally worse off while neutralization is the best of possible outcomes; an ideal war robot ordered to kill could refuse and instead apprehend, analogous to military ethics rules against obeying orders to commit war crimes.

    “neutralizing the threat is better than just say killing the agent, because that would be certainly making the agent now worst off. But neutralizing the threat would be the best of all possible outcomes.”

  • Because neither human drivers nor AI can predict outcomes with certainty, self-driving cars will constantly make decisions between higher-risk and lower-risk actions even when actual collisions are rare, so likelihood-based path pruning (as in model predictive control) must be supplemented by weighting the severity of likely collisions—a likely collision with a pedestrian is worse than one with a tree.

    “I think we also need to say a likely collision with a pedestrian is worse than a likely collision with a tree.”

  • Under contractarianism, a car may evaluate a path toward a helmeted cyclist as 'better' (less harmful) than one toward an unhelmeted cyclist, and may prefer colliding with a more survivable adult over a more vulnerable baby—not as punishment or preference, but because the language of targeting and punishing is misleading; the car only judges one path's worst collision as less severe.

    “the path that leads to you was evaluated as better than the path that led to her.”

  • Maximin implies it would be better for a car to swerve into a crowd of 50 people giving each a single broken leg than to hit a wall giving the passenger two broken legs, because fifty single broken legs each is a better worst-case than one instance of two broken legs—a prediction Leben finds crazy but accepts as a consequence of his theory.

    “it would be better for the car to swerve into a crowd of 50 people and give them all a broken leg, rather than to swerve into a brick wall, and give the passenger in the vehicle, two broken legs. Why is that? Because 50 single broken legs, is better than one instance of two broken legs.”

  • As autonomous technologies in transportation, medicine, and warfare make decisions affecting human well-being without human supervision, we inevitably must decide what rules to program into them; every programming choice reveals normative assumptions, making it impossible to stay free of ethics.

    “It's sort of impossible to stay free of ethics, to just say, "Well, I'm going to avoid ethics entirely."”

  • Leben does not call for banning autonomous systems in war outright, but holds all his claims are hypothetical—if we build such machines, here are the capabilities they must have—and he is more skeptical that military robots will achieve the required sophistication than medical or transportation systems, acknowledging the charge that this conditional stance may be too corporate.

    “all of my claims are a big hypothetical, which is if we are going to design these kinds of machines, these are the kinds of capabilities that they would require.”

  • Contractarianism, like most moral theories, holds that information such as a person's gender, profession, religion, or social status is irrelevant to collision decisions; only factors bearing on the physical effects of the collision—physical position, size, and possibly age—are relevant, which helps distinguish discriminatory databases from non-discriminatory ones.

    “whether a person is a doctor or a lawyer, whether a person is a Muslim or a Christian or an atheist, all of that is irrelevant. But what is relevant, are things like your physical position, your physical size, and maybe your age”

  • While the best approach is implementing maximin top-down so a system can articulate its reasoning, a bottom-up machine-learning system that reliably produces outputs matching maximin and that we have good reason to expect will continue to do so would be close to good enough, so AI need not be an articulate moral philosopher.

    “If the machine learning system produced outputs that always matched the maximin principle in the kinds of cases we observe, and we had good reason for thinking that it would continue to run this program that approximated maximin in future cases. Then I would say that would be... close to good enough”

  • Car companies' official position that all collisions are equally bad and should be avoided equally is untenable because collisions plainly differ in severity (being hit at two mph versus twenty mph), so vehicles evaluating paths must judge some collisions as better than others.

    “a lot of the major car companies have the official position of just saying, "Well, we think all collisions are bad and we want to avoid them all equally," but I think that's an incredibly ridiculous position to take because not all collisions are equal.”

0.73

Contractarian maximin better explains moral behavior and produces cooperation than utilitarian summation across game-theoretic and real-world scenarios.

6 pointscentrality 5/5
  • Unlike utilitarians who care about suffering and happiness, contractarians care only about the distribution of primary goods—health, safety, essential resources—aiming to bring the worst-off up to a minimal threshold of normal functioning and opportunity, not to maximize happiness.

    “if I'm a contractarian, I don't care about their suffering. I care about the distribution of primary goods, namely their health, their safety, their essential resources.”

  • Maximin is superior to utilitarian summation because in cases like the repugnant prisoner's dilemma or utility monster, summation would endorse making everyone slightly worse off to greatly benefit one person, whereas maximin protects the worst-off; the real motivation for maximin is that it produces cooperative behavior in all scenarios.

    “you could arrange scenarios where the sum of all the pay-off for people is actually not either what maximin would say nor is it what Pareto optimality would predict over just self-interested behavior. And for that reason, I think that the maximin principle is actually the better principle than the utilitarian one.”

  • Cooperation problems can be defined formally as situations where self-interested behavior (a Nash equilibrium, where no one can unilaterally improve without hurting another) leaves available Pareto improvements (changes that make everyone better off or at least no one worse off) that self-interested rational agents cannot reach on their own.

    “I'm defining cooperation problems as ones where self-interest here, and self-interest, I'm measuring as a Nash equilibrium... But there exist Pareto improvements, from Nash equilibria.”

  • The maximin principle, drawn from game theory, says we should prefer the distribution that makes the worst-off person as well off as possible: attach values to each outcome, identify the worst-off in each, and select the option with the highest of those lows.

    “The maximin principle says that we should prefer the distribution that makes the worst off person as best off as possible.”

  • A utilitarian or contractarian playing repeated prisoner's dilemmas will get better outcomes over time than a defector—measured in money you will make more, measured in children you will have more—because cooperation is favored over iteration.

    “If you are a utilitarian, or a contractarian playing prisoner's dilemma, over and over and over again, you will get better outcomes, if you're measuring this in money you'll make more money over time. If you're measuring this in children, you'll have more children over time.”

  • Because moral theories evolved for the purpose of promoting cooperative behavior among self-interested organisms, the way to adjudicate between equally internally-consistent moral theories is to ask which one actually produces more cooperative behavior; that is the one we should use.

    “Which moral theory actually fulfills this meta-ethical assumption that moral theories evolve for the purpose of promoting cooperative behavior among self-interested organisms? If you have a moral theory that actually is better at creating cooperative behavior than other kinds of theories, then that's the one we should use”

0.50

Moral intuitions are unreliable guides; people's explicit moral theories diverge wildly and their actual choices contradict their stated principles.

3 pointscentrality 4/5
  • Leben gives moral intuitions virtually no evidential weight because of the long history of strong intuitions that turned out false, and notes many of his current intuitions (loving meat, driving fossil-fuel cars) are likely judged wrong by almost any plausible moral theory.

    “I'm so aware of the history of strong intuitions that have been false. That I just give them virtually no evidential weight whatsoever.”

  • A PhilPapers survey of professional philosophers shows roughly 25% lean deontology, ~23% consequentialism, 18% virtue ethics, and 32% other, while on metaethics 56.4% lean moral realism, 27.7% anti-realism, and 15% other—an embarrassing lack of consensus given these theories materially affect how we live and design AI.

    “25% accept or lean deontology, 23% and some change accept or lean towards consequentialism, 18% towards virtue ethics and 32.3% other.”

  • Many well-educated people are explicitly moral relativists but implicitly utilitarians, willing to sacrifice one to save many yet unwilling to give up cappuccinos, fossil fuel, or meat, and falling back on relativism only when it is convenient.

    “they are explicitly moral relativists, but implicitly utilitarians.”

0.44

Moral facts are objective as hypothetical imperatives tied to shared human goals, grounded in contingent cooperative purposes, not absolute truth.

1 pointcentrality 4/5
  • Moral facts are objective in the way that 'smoking is bad for your health' is objective—they are real but contingent on goals we share; framed as hypothetical imperatives (if you want X, do Y), there are objective answers within the realm of shared goals, but outside those shared goals no state can be called better or worse.

    “we can have objective answers to this question within the domain of us all sharing these goals”

0.36

Machine learning systems copying human moral behavior will replicate human biases; abstract principles are necessary to avoid racist and sexist outcomes.

3 pointscentrality 3/5
  • Rawls's original position imagines designing society without knowing who you will be, so you agree on distributions of primary goods—life, health, opportunity, essential resources—which all humans must value to pursue any goal, as distinct from secondary goods (like coffee or television) that depend on contingent preferences.

    “Primary goods are the kinds of things that all human beings value no matter what; that all human beings have to value in order to pursue any kind of goal at all.”

  • Training machines to imitate human moral behavior in a bottom-up approach will incorporate the biases and limitations of the humans modeled, producing machines that are racist, sexist, and poor at reasoning about consequences versus rewards.

    “you're probably going to incorporate all of the terrible biases and limitations of the humans that you're using to model your machine's behavior on. So these machines are probably going to wind up being terribly racist and sexist”

  • Trolley-problem-type dilemmas, where one must choose between causing harm to one person or allowing harm to many, occur frequently in real professions like medicine, warfare, and business, and Truman's decision to drop the atomic bomb on Japan was structurally a trolley problem.

    “they happen quite often in certain professions, like in medicine, and warfare and business, where you have to make choices about causing harm to one person or allowing harm to many other people.”

0.34

Trolley problems reveal that people track non-consequentialist factors like physical intrusion and genetic relatedness, not just outcomes.

2 pointscentrality 3/5
  • Most people judge it permissible to divert a trolley onto a single person to save five but impermissible to push a large man to his death to save five, even though the consequences are identical, revealing that people's intuitions track factors like physical intrusion, causal chains, or intentions rather than pure consequences.

    “it turns out that most people say it's permissible in surveys to switch the track onto the single person, but not permissible to push the large man to his death, even though that seems strangely inconsistent.”

  • An experiment by April Bleske-Rechek found that varying the genetic relatedness of the person on the side track alters whether people will pull the switch, almost exactly in proportion to genetic relatedness (brother, cousin, etc.), even though people explicitly deny that relatives are more valuable.

    “they found that alternating by condition, the genetic relatedness of the person on the side track, will alternate whether people are willing to pull the switch, almost exactly as you would predict by the proportion of their genetic relatedness.”