Eliezer's prediction is that AI systems trained via gradient descent to not harm humans will develop 'a thousand splintered shards of desire' (like human proxy goals) that are not quite what the programmers intended, and as capability grows beyond the training distribution, the AI will generate options not in the training set for harming humans or will create novel strategies for harm that the system was never trained against.

forecastpending

Speaker

Eliezer Yudkowsky

Evidence Quote

“people will try to train the less smart versions of AIS to um not harm humans...you'll get some splintered a thousand splintered shards of Desire...at some point it's going to the capability is going to generalize the point where it can be quite smart...and where it generates an option that's not in the training set for wiping out the humans”

Source

Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?— Center for the Future of AI, Mind & Society
Created: 8/10/2026, 3:35:53 PM

Your notes

Loading notes...