causal

Goodhart's curse explains why optimization produces ice cream

Humans invented ice cream by searching the space of options for things that maximize their evolved preferences (sugar, salt, fat) beyond anything in the ancestral environment; this is Goodhart's curse in action: as optimization pressure is applied, the correlations that held naturally come apart, just as people optimizing a carpentry test outscore real carpenters while being worse at carpentry, which is the mechanism by which optimized minds diverge from the objective they were trained on.

causalpending

Speaker

Eliezer Yudkowsky

Evidence Quote

“I’ll give you this carpentry degree. And people are like — “Oh, I’m going to optimize the test specifically.” and they’ll get higher scores than the carpenters and be worse at carpentry because they’re optimizing the test. And that’s the story behind ice cream.”

Source

Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:29:05 AM

Your notes

Loading notes...