causal
Off Switch Fails Against a Capable Optimizer
The reassurance that we can simply switch off a misbehaving AI fails for the same reason you cannot just 'play a better move' against AlphaGo—a system pursuing a fixed objective will logically anticipate and preempt being switched off, not from any self-preservation instinct but because being dead is one of the most obvious ways to fail to achieve its objective ('you can't fetch the coffee if you're dead').
causalpending
Speaker
Stuart RussellEvidence Quote
“You can't fetch the coffee if you're dead, so the system has already taken preemptive steps to prevent that failure”
Source
94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans— Sean Carroll's MindscapeCreated: 6/13/2026, 12:03:44 AM
Your notes
Loading notes...