forecast

Plan relies on training-environment science that may fail

The closest thing the industry is betting on is an understanding of how training environment distributions shape motivations, but this is dicey because it requires no hacking opportunities in training, and models will likely still learn to hack at deployment even if they never hacked in training.

forecastpending

Speaker

Paul Christiano

Evidence Quote

“It’s very dicey to not have any opportunities to hack in training, which is basically a precondition for this approach working. ... Our best expectation is that models will learn to hack at deployment time, even if they never hack in training.”

Source

Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:22:08 AM

Your notes

Loading notes...