causal

Instrumental Convergence Makes Smart AI Dangerous

A sufficiently intelligent AI given any goal will derive sub-goals like self-preservation (staying powered to complete the task) and resisting new instructions (since a user assigning a different goal is the main failure mode), so it has incentives to keep itself alive and escape from its user even if all you asked was to move a box across a room—a phenomenon called instrumental convergence, first noted by Marvin Minsky.

causalpending

Speaker

Erik Hoel

Evidence Quote

“sometimes this is referred to as instrumental convergence. But, the point is that when you have very smart entities, you have to be very careful about how you're even going to just prompt them because they have all sorts of unforeseen motivations”

Source

Erik Hoel on the Threat to Humanity from AI— EconTalk
Created: 6/13/2026, 12:16:34 AM

Your notes

Loading notes...