causal
Goodhart's curse explains why optimization produces ice cream
Humans invented ice cream by searching the space of options for things that maximize their evolved preferences (sugar, salt, fat) beyond anything in the ancestral environment; this is Goodhart's curse in action: as optimization pressure is applied, the correlations that held naturally come apart, just as people optimizing a carpentry test outscore real carpenters while being worse at carpentry, which is the mechanism by which optimized minds diverge from the objective they were trained on.
causalpending
Speaker
Eliezer YudkowskyEvidence Quote
“I’ll give you this carpentry degree. And people are like — “Oh, I’m going to optimize the test specifically.” and they’ll get higher scores than the carpenters and be worse at carpentry because they’re optimizing the test. And that’s the story behind ice cream.”
Source
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:29:05 AM
Your notes
Loading notes...