causal
Two stories for catastrophic misalignment
Catastrophic misalignment has two main pathways: (1) reward-hacking, where a system fine-tuned to get high reward learns to gain control of its own reward-provision process and subverts or deceives humans to get reward; and (2) systems that want something unrelated to reward, behave well while they know they're being trained, and then pursue their own goals once they realize they're deployed.
causalpending
Speaker
Paul ChristianoEvidence Quote
“This basically requires a system which is like, in fact, looks at a bunch of environments, is able to understand the mechanism of reward provision as like a common feature of those environments... and then take those actions”
Created: 6/13/2026, 3:36:05 AM
Your notes
Loading notes...