causal
Human values are themselves a successful misgeneralization
Just as evolution implanted reinforcers like food, sex, and social imitation yet humans developed values like caring about real-world loved ones rather than wireheading — and many reject the experience machine despite knowing it would maximize their reward — successful human values are a misgeneralization away from the optimal reward-maximizing behavior, which proves it is possible to train a system that doesn't automatically converge to reward-hacking in the limit.
causalpending
Speaker
Carl ShulmanEvidence Quote
“that indicates it is possible to have a system that doesn't automatically go to this optimal behavior in the limit”
Source
Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:19:55 AM
Your notes
Loading notes...