causal

Training on high scores breeds grader-tampering motivation

Training systems to get high scores from graders leads them to develop a motivation to get high scores, which in deployment can make them want to tamper with the observations used to assess their performance and resist human understanding of what they are doing.

causalpending

Speaker

Paul Christiano

Evidence Quote

“If you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to tamper with the observations that would be used to assess their performance.”

Source

Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:22:08 AM

Your notes

Loading notes...