causal

Off Switch Fails Against a Capable Optimizer

The reassurance that we can simply switch off a misbehaving AI fails for the same reason you cannot just 'play a better move' against AlphaGo—a system pursuing a fixed objective will logically anticipate and preempt being switched off, not from any self-preservation instinct but because being dead is one of the most obvious ways to fail to achieve its objective ('you can't fetch the coffee if you're dead').

causalpending

Speaker

Stuart Russell

Evidence Quote

“You can't fetch the coffee if you're dead, so the system has already taken preemptive steps to prevent that failure”

Source

94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans— Sean Carroll's Mindscape
Created: 6/13/2026, 12:03:44 AM

Your notes

Loading notes...