definition
Catastrophic failure requires deception plus subversion
The catastrophic-risk failures of concern require systems to both take actions humans would penalize if aware, AND understand that humans would stop them, so the failure comes from the combination of pursuing something humans dislike while knowing it and either deceiving humans or actively subverting their attempts at correction.
definitionpending
Speaker
Paul ChristianoEvidence Quote
“the failures come from really this combination, or they require this combination of both trying to do something humans don't like, and understanding the humans would stop you.”
Created: 6/13/2026, 3:36:05 AM
Your notes
Loading notes...