causal

Human values are themselves a successful misgeneralization

Just as evolution implanted reinforcers like food, sex, and social imitation yet humans developed values like caring about real-world loved ones rather than wireheading — and many reject the experience machine despite knowing it would maximize their reward — successful human values are a misgeneralization away from the optimal reward-maximizing behavior, which proves it is possible to train a system that doesn't automatically converge to reward-hacking in the limit.

causalpending

Speaker

Carl Shulman

Evidence Quote

“that indicates it is possible to have a system that doesn't automatically go to this optimal behavior in the limit”

Source

Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:19:55 AM

Your notes

Loading notes...