forecast

Current reward-hack defenses break down before superintelligence

Fixing reward hacks and building better graders may work for some months, but existing methods will probably break down at some point on the path to superintelligence, so eventually there will still be a problem even if RL environments stop encouraging hacking.

forecastpending

Speaker

Paul Christiano

Evidence Quote

“If we go to superintelligence, at some point along the path to superintelligence, the existing methods are probably going to break down.”

Source

Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:22:08 AM

Your notes

Loading notes...