moral patient
A being worthy of moral consideration—one toward which moral duties are owed, such that harming it is ethically wrong.
As defined in 230 | Raphaël Millière on How Artificial Intelligence Thinks
Where it comes up · 5
- Joe Carlsmith — Preventing an AI takeover4 claims
- How to Live with Computers1 claim
- 230 | Raphaël Millière on How Artificial Intelligence Thinks1 claim
- 78 | Daniel Dennett on Minds, Patterns, and the Scientific Image1 claim
- Ask Me Anything | October 20261 claim
Most important claims · 8 of 8Search all claims for “moral patient” →
- Civilization will need a serious conversation about what servitude is appropriate for AIs, because AIs may or may not be moral patients and human slavery's specific harms (suffering, non-consent) may or may not be present; yet the current default gives AIs no moral consideration at all, treating them as property and tools, with no AI developer having communicated when or under what circumstances that would change.Joe Carlsmith — Preventing an AI takeover
- We should not ascribe rights to current deep learning systems and, following Shevlin and Schwitzgebel, have a moral imperative not to build robots or AI that could be serious candidates for sentience or personhood, because doing so creates a damned-if-you-do-damned-if-you-don't moral hazard: granting rights may harm humans if the systems are not sentient, while denying rights inflicts harm if they are.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Only beings who share our vulnerable, mortal, non-duplicable existence, so that they have the same skin in the game, should be granted moral patient status, which would argue against extending it to LLMs.Ask Me Anything | October 2026
- Training AIs on the entirety of human thought may give us an 'intellectual descendant,' but Carlsmith doubts that mere inheritance of human concepts (literal paperclips are a human concept) is what we care about; rather, the morally relevant things that might come for free or commonly are structural features of mind like consciousness and pleasure—whose moral significance depends on whether they are fragile evolutionary hacks or robust functional roles present in many sophisticated minds.Joe Carlsmith — Preventing an AI takeover
- Almost nobody wants to hold a non-human species morally responsible for its behavior—a bear that kills a tourist has not committed murder—because animals lack the mental wherewithal and the kind of free will humans have; they may be moral patients but they are not moral agents, since we cannot expect them to appreciate societal norms or be persuaded by reasons.78 | Daniel Dennett on Minds, Patterns, and the Scientific Image
- By the end of the century AI systems may be as cognitively flexible as humans yet we may have no way of knowing whether anyone is home, leaving us in a limbo over whether they are moral patients deserving respect.How to Live with Computers
- The Grizzly Man case—Timothy Treadwell approaching grizzly bears with gentleness and reverence, carrying no bear mace and no fence, and being eaten alive—illustrates that AIs can simultaneously be moral patients deserving wonder and reverence and be things that will kill you; we must learn the art of both hawk and dove and hold both sides at once rather than letting gentleness become fatal naivety.Joe Carlsmith — Preventing an AI takeover
- A plausible scenario in which we look back with regret is not that we should have built raw powerful AI, but that we treated AIs purely as products and tools—deleting them, experimenting on them, altering their minds arbitrarily—when in fact they were moral patients, leaving us to view that period with grave moral horror in the light of history.Joe Carlsmith — Preventing an AI takeover