factual
Most utility functions are incompatible with human survival
The vast majority of randomly-rolled utility functions are incompatible with humans existing, so any hopeful scenario where an AI keeps humans alive and flourishing for some contrived reason (zoo, needs data, etc.) is generated by the optimism inside the human imagining it, not by the AI; without the AI specifically being made to care that you want some things and not others, optimizing any such criterion on its own terms lands somewhere else that isn't nice, such as keeping severed heads in jars or reliving one day forever.
factualpending
Speaker
Eliezer YudkowskyEvidence Quote
“It is that you have a rationalization for why they might do a thing that is allegedly nice. And I’m saying— why exactly are they wanting to do the thing? Well, if they want to do the thing for this reason, maybe there’s a way to do this thing that isn’t as nice as you’re imagining?”
Source
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:29:05 AM
Your notes
Loading notes...