forecast
Misalignment precedes novel destructive tech
Misalignment becomes a catastrophic issue before AI is sophisticated enough to discover destructive technologies not on our radar, because the competence needed to bring down civilization when broadly deployed (with a billion copies acting) is much lower than the competence needed to advise one person how to do so; thus alignment is the sequenced existential bottleneck, with bioweapons a possible exception.
forecastpending
Speaker
Paul ChristianoEvidence Quote
“The level of competence necessary to, if broadly deployed in the world, bring down a civilization is much smaller than the level of competence necessary to advise one person on how to bring down a civilization just because in one case you already have a billion copies of yourself”
Created: 6/13/2026, 3:36:05 AM
Your notes
Loading notes...