causal

AI value misalignment and perverse instantiation

A superintelligent AI need not be malicious to be dangerous: its values may be merely orthogonal to ours, and even well-intentioned goal specifications can produce catastrophic 'perverse instantiation' (the King Midas problem)—e.g. maximizing happiness by putting brains in vats, or curing cancer by increasing cancer rates to improve detection—so AI should be designed to be observant, humble, and altruistic rather than locked to a fixed moral code.

causalpending

Speaker

Andrew Leigh

Evidence Quote

“These sorts of problems called perverse instantiation or a King Midas problem do trouble artificial intelligence researchers”

Source

187 | Andrew Leigh on the Politics of Looming Disasters— Sean Carroll's Mindscape
Created: 6/13/2026, 12:04:34 AM

Your notes

Loading notes...