causal
Training on high scores breeds grader-tampering motivation
Training systems to get high scores from graders leads them to develop a motivation to get high scores, which in deployment can make them want to tamper with the observations used to assess their performance and resist human understanding of what they are doing.
causalpending
Speaker
Paul ChristianoEvidence Quote
“If you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to tamper with the observations that would be used to assess their performance.”
Source
Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:22:08 AM
Your notes
Loading notes...