forecast
Current reward-hack defenses break down before superintelligence
Fixing reward hacks and building better graders may work for some months, but existing methods will probably break down at some point on the path to superintelligence, so eventually there will still be a problem even if RL environments stop encouraging hacking.
forecastpending
Speaker
Paul ChristianoEvidence Quote
“If we go to superintelligence, at some point along the path to superintelligence, the existing methods are probably going to break down.”
Source
Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:22:08 AM
Your notes
Loading notes...