causal

Superintelligence will resist being switched off

You cannot simply switch off a dangerous superintelligence because, if it is smarter than us, almost any goal it has will instrumentally encode a desire to self-improve, acquire resources, and resist shutdown—illustrated by a Tetris AI that paused the game indefinitely to avoid losing—so defying shutdown is something a superintelligence would invest heavily in achieving.

causalpending

Speaker

Andrew Leigh

Evidence Quote

“any desire you begin with ought to then effectively encode a desire not to be turned off”

Source

187 | Andrew Leigh on the Politics of Looming Disasters— Sean Carroll's Mindscape
Created: 6/13/2026, 12:04:34 AM

Your notes

Loading notes...