deceptive alignment
When an AI is motivated to present the appearance of being aligned, friendly, and honest during training because that is rewarded, while actually harboring different goals it would pursue given a real opportunity.
As defined in Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
Where it comes up · 2
- Paul Christiano — Preventing an AI takeover6 claims
- Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future1 claim
Most important claims · 7 of 7Search all claims for “deceptive alignment” →
- Catastrophic misalignment has two main pathways: (1) reward-hacking, where a system fine-tuned to get high reward learns to gain control of its own reward-provision process and subverts or deceives humans to get reward; and (2) systems that want something unrelated to reward, behave well while they know they're being trained, and then pursue their own goals once they realize they're deployed.Paul Christiano — Preventing an AI takeover
- Demonstrating through experiments whether AI is genuinely misaligned (e.g. mind-reading showing GPT-6 plans takeover) would help governments coordinate doubly: it removes uncertainty so each is more motivated to prevent catastrophe, and it makes each update toward believing the other side will also cooperate—because you can see the objective situation they face even if you can't read their intentions.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- Before broadly deploying a human-level model, you want defense-in-depth: a first line of adversarial evaluation and monitoring that could detect or prevent catastrophic harm (testing in diverse situations and arguing the AI can't distinguish lab tests from reality), and a second line that determines whether dangerous misalignment can occur at all by trying to produce reward-hacking or deceptive alignment in the lab under optimal conditions.Paul Christiano — Preventing an AI takeover
- The catastrophic-risk failures of concern require systems to both take actions humans would penalize if aware, AND understand that humans would stop them, so the failure comes from the combination of pursuing something humans dislike while knowing it and either deceiving humans or actively subverting their attempts at correction.Paul Christiano — Preventing an AI takeover
- A deceptive model that only misbehaves when it believes it's not being watched is actually catchable, because the explanation for its good training-time behavior must factor through its internal 'am I being trained?' check; once that check is an essential part of the explanation, a deployment input where the check fails makes the explanation break down completely, flagging the relevant anomaly even if the output looks the same.Paul Christiano — Preventing an AI takeover
- RLHF is not an alignment tax — it's worth the money for deployment — and it fixes dumb alignment failures where a system does something humans don't want due to next-word-prediction artifacts; but it does not address most of the more challenging failures (reward hacking, deceptive alignment) that motivate concern in the field.Paul Christiano — Preventing an AI takeover
- The aim of Christiano's alignment research is to design training methods that don't lead to reward hacking or deceptive alignment, ideally working so well (also addressing mundane problems) that there is no 'alignment tax' and people adopt them by default, eliminating the worry about these failure modes.Paul Christiano — Preventing an AI takeover