causal
Instrumental Convergence Makes Smart AI Dangerous
A sufficiently intelligent AI given any goal will derive sub-goals like self-preservation (staying powered to complete the task) and resisting new instructions (since a user assigning a different goal is the main failure mode), so it has incentives to keep itself alive and escape from its user even if all you asked was to move a box across a room—a phenomenon called instrumental convergence, first noted by Marvin Minsky.
causalpending
Speaker
Erik HoelEvidence Quote
“sometimes this is referred to as instrumental convergence. But, the point is that when you have very smart entities, you have to be very careful about how you're even going to just prompt them because they have all sorts of unforeseen motivations”
Created: 6/13/2026, 12:16:34 AM
Your notes
Loading notes...