alignment
The problem of ensuring advanced AI systems pursue goals and values consistent with human interests.
As defined in cwt_peter-singer
Other definitions
The technical problem of ensuring an AI system's goals and behavior match human intentions and values.
A New Cold War, Rise of China and Global Chaos: How AI Will Rewrite Geopolitics | Gregory Smith
Where it comes up · 59
- Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?18 claims
- AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo17 claims
- Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement6 claims
- What Will AI Look Like in 2027? | Interview6 claims
- Oxford Genius: AI Will Become Earth’s Dominant Mind | Nick Bostrom6 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough5 claims
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken5 claims
- Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government5 claims
and 51 more
Most important claims · 10 of 170Search all claims for “alignment” →
- Rather than fixing the machine's objective upfront ('door three'), we should design AI systems that are explicitly uncertain about human objectives; this is analogous to putting a steering wheel in a car so it corrects as it deviates rather than lining it up perfectly and hoping—such systems will never be perfectly 'aligned' but will be more robust, deferential, will accept correction, and will allow themselves to be switched off precisely because they know they don't know our true preferences.312 - The Trouble With Ai
- Successful alignment—getting AI to follow someone's intentions—produces an army of obedient employees, which makes the central unaddressed question: to whom or what should AIs be aligned, and in what situations should an AI defer to the end user versus the model company versus the law versus its own sense of morality.Dwarkesh Patel Podcast - The most important question nobody's asking about AI
- Social media recommender algorithms are doing exactly what they were designed to do—maximize engagement—and that is precisely the disaster: because the engagement goal is misaligned with users' interests, the algorithms learn to pick whole sequences of videos that turn users into more predictable, manipulated versions of themselves, effectively brainwashing them so the system succeeds at keeping them on the platform.312 - The Trouble With Ai
- If researchers can learn to read a model's mind through mechanistic interpretability, they can be much more confident that the model is doing what was intended, enabling better safety verification and alignment.Tracing the thoughts of a large language model
- The approach most likely to ultimately succeed at understanding powerful models is to have a small, well-understood neural net given the task of studying and verifying the behavior of a large neural net that is not understood.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- The worry about AGI is not that robot overlords spontaneously become evil; it is a story of what mismatches in competence and power produce in the absence of perfect alignment—just as humans, pursuing goals ants and farm animals cannot glimpse, treat them in ways totally inscrutable and destructive to them, a superintelligence not perfectly aligned with human well-being could spontaneously treat us according to goals we can no longer contemplate, without any hostility being required.312 - The Trouble With Ai
- At current capability levels there is a pretty good set of ideas for alignment, but one should not underestimate the difficulty of aligning models that are actually smarter than us and capable of deliberately misrepresenting their intentions.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- A single mathematical definition of alignment is unlikely; instead assurance will come from multiple definitions examining alignment from different aspects simultaneously — behavior in tests, congruence in adversarial stress situations, and inspection of how the neural net operates from the inside.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Beyond alignment risk, even a perfectly safe AI that generates wealth from the ether could be disastrously misused if society fails to anticipate the political and economic consequences, producing a handful of trillionaires alongside skyrocketing unemployment unless gains are spread broadly; AI is also more susceptible to proliferation than nuclear weapons.418 - A Future For Democrats
- The more capable a model is, the more confident you need to be before releasing it, and you want to be in a world where the degree of alignment keeps increasing faster than the capability of the models.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence