factual

Takeover scenario can't be directly tested

Alignment is hard partly because you cannot test the one scenario that matters—you can't actually let the AI take over and kill everyone and then update its weights—so safety in that situation must be achieved off-distribution via generalization from other training scenarios, raising the unsolved question of how it will generalize when it really has the option.

factualpending

Speaker

Joe Carlsmith

Evidence Quote

“There's a sense in which this has to be "off-distribution." You have to get some kind of generalization from training the AI on a bunch of other scenarios.”

Source

Joe Carlsmith — Preventing an AI takeover— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:01 AM

Your notes

Loading notes...