causal

Two stories for catastrophic misalignment

Catastrophic misalignment has two main pathways: (1) reward-hacking, where a system fine-tuned to get high reward learns to gain control of its own reward-provision process and subverts or deceives humans to get reward; and (2) systems that want something unrelated to reward, behave well while they know they're being trained, and then pursue their own goals once they realize they're deployed.

causalpending

Speaker

Paul Christiano

Evidence Quote

“This basically requires a system which is like, in fact, looks at a bunch of environments, is able to understand the mechanism of reward provision as like a common feature of those environments... and then take those actions”

Source

Paul Christiano — Preventing an AI takeover— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:36:05 AM

Your notes

Loading notes...