definition

Catastrophic failure requires deception plus subversion

The catastrophic-risk failures of concern require systems to both take actions humans would penalize if aware, AND understand that humans would stop them, so the failure comes from the combination of pursuing something humans dislike while knowing it and either deceiving humans or actively subverting their attempts at correction.

definitionpending

Speaker

Paul Christiano

Evidence Quote

“the failures come from really this combination, or they require this combination of both trying to do something humans don't like, and understanding the humans would stop you.”

Source

Paul Christiano — Preventing an AI takeover— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:36:05 AM

Your notes

Loading notes...