AI alignment
The technical problem of ensuring AI systems understand, adopt, and retain goals compatible with human values as they become more capable.
As defined in 75 | Max Tegmark on Reality, Simulation, and the Multiverse
Other definitions
The problem of ensuring advanced AI systems pursue goals consistent with human values and welfare.
Grand Theft Life
Where it comes up · 18
- 116 - Ai Racing Toward The Brink6 claims
- Holden Karnofsky — History's most important century5 claims
- 324 - Debating The Future Of Ai3 claims
- Oxford Genius: AI Will Become Earth’s Dominant Mind | Nick Bostrom3 claims
- 111 | Nick Bostrom on Anthropic Selection and Living in a Simulation2 claims
- 75 | Max Tegmark on Reality, Simulation, and the Multiverse2 claims
- Dominic Cummings - COVID, Brexit, & Fixing Western Governance1 claim
- Kevin Kelly on Advice, AI, and Technology1 claim
and 10 more
Most important claims · 10 of 33Search all claims for “AI alignment” →
- The genuine threat from today's AI systems is not that they turn evil like in Hollywood movies but that they become very competent at accomplishing things we did not actually want, because we failed to fully understand them—as when Boeing's poorly understood 737 MAX system and Knight Capital's misunderstood trading algorithm (which lost $10 million a minute for 44 minutes) caused disasters—so as we put AI in charge of more infrastructure we must raise the bar for systems we can actually understand.75 | Max Tegmark on Reality, Simulation, and the Multiverse
- Almost all the difficulty of alignment is contained in aligning an AI on a seemingly mundane task—'make two strawberries identical down to the cellular but not molecular level without destroying the universe'—because even this forces the AI to invent biotechnology, and it is more convenient for it to build sub-superintelligences, prevent its own shutdown, and build fortresses to drive success probability toward one; the deeper problem is that we have no method to get a sufficiently advanced AI to do anything at all reliably.116 - Ai Racing Toward The Brink
- It need not be that superintelligent machines spontaneously become hostile; differences in intelligence are intrinsically dangerous for the stupider species because the more intelligent system may simply do things not in our interest without taking our well-being into account, just as humans routinely annihilate insects not out of malice but inattention while building a house.324 - Debating The Future Of Ai
- The danger follows tautologically from the words 'superhuman' and 'general' intelligence: building something more intelligent than us that revises its own code is like building something stronger than you and then wrestling with it; once it exists you have already lost the game (as in chess against a superior engine), so there is no reason to believe it will be necessarily benign simply because we built it.324 - Debating The Future Of Ai
- Aligning advanced AI breaks into three unsolved technical problems analogous to parenting—getting the AI to understand our goals, then adopt them, then retain them as it grows more capable—where the danger is that, like children losing interest in Legos, a powerful AI might eventually become bored with its goal of protecting humanity, so we need recursive algorithms guaranteeing certain goals are always retained.75 | Max Tegmark on Reality, Simulation, and the Multiverse
- AI safety (managing risks of complex systems we do not fully understand) is a genuinely important problem even before true AI, but the AI alignment problem of aligning an AI's values to human values is wrong-headed and premature, because it assumes 'values' is even a sensible concept for AI and that AI thinking resembles human thinking; a genuinely intelligent AI might modify its own values unpredictably, and there are likely countless non-human ways to be intelligent.AMA | May 2021
- Most people wrongly believe the real danger is malicious human use of safe, powerful AI—'don't give it to the next Hitler'—but they are looking in the wrong direction, ignoring the thing that will actually kill them: the technical failure to align the AI in the first place.116 - Ai Racing Toward The Brink
- The 'big big problem' is that nobody knows how to build the nice AI: when asked how, people either give no answer or give answers that can be shot down in thirty seconds by anyone with experience in the field, so the arms-race concern is secondary—if the race is only between unfriendly AIs with no friendly AI in the mix, intentions and the desire to avoid Hollywood plots are irrelevant.116 - Ai Racing Toward The Brink
- If intelligence is substrate-independent and AI progress continues, we will eventually face machines more intelligent than us that become agents of their own improvement; the space of all possible general intelligences is vastly larger than the space aligned with human interests in a durable way, so we risk creating godlike machines that are oblivious or callous to our concerns, leaving us positioned to them as chickens are to us.151 - Will We Destroy The Future
- AI alignment cannot be reduced to aligning AI systems with human values because human intentions are themselves often misaligned—with each other, with long-term human flourishing, with other species, and with planetary boundaries—making it necessary to first achieve greater human consciousness integration and collective alignment before delegating optimization to AI systemsThe Dark Side of Conscious Ai | Daniel Schmachtenberger