causal

Optimizing for X produces a mind that does not explicitly want X

When you take an optimizer like natural selection or gradient descent and optimize a complex system hard enough for X that it becomes a general intelligence doing X, the resulting intelligence has no explicit goal of X: natural selection hill-climbed humans for inclusive genetic fitness, yet humans have hundreds of small goals and none is fitness maximization, and the same instability holds for any current method of producing general intelligence.

causalpending

Speaker

Eliezer Yudkowsky

Evidence Quote

“we have no explicit goal of doing fitness maximization we have hundreds of of of different little goals none of them are the thing that natural selection was hill climbing us to do”

Source

116 - Ai Racing Toward The Brink— Making Sense with Sam Harris
Created: 6/14/2026, 2:07:58 AM

Your notes

Loading notes...