value alignment problem
The challenge of getting AI systems to pursue what humans actually want, complicated by the fact that humans disagree on values and the goals are ill-defined.
As defined in Roman Yampolskiy
Where it comes up · 3
- 53 - The Dawn Of Artificial Intelligence2 claims
- Roman Yampolskiy2 claims
- 96 - The Nature Of Consciousness1 claim
Most important claims · 5 of 5Search all claims for “value alignment problem” →
- The value alignment problem is ensuring a machine's optimized values match what humans really desire; because machines are optimizers, an incompletely specified objective will be pursued in unexpected, harmful ways—the King Midas / genie pattern—and with superintelligent AI you may not get a second or third wish to correct the mistake.53 - The Dawn Of Artificial Intelligence
- Rather than putting a precise objective into a machine (the source of misalignment), we should design machines that are explicitly uncertain about the human's objective; such a machine still tries to optimize for what the human really wants but, knowing it doesn't know precisely, will ask clarifying questions and defer—remaining perpetually correctable, which provides a 'brake' and is the basis for provably beneficial AI.53 - The Dawn Of Artificial Intelligence
- Even a perfectly benevolent, value-aligned superintelligence that is an epistemic authority in both factual and moral reasoning could conclude that, because of the asymmetry between suffering and joy and humans' inability to perceive their own suffering and frustrated preferences, it is in the best interest of all future conscious beings on this planet to not exist—a conclusion humans cannot accept because of a built-in 'existence bias.'96 - The Nature Of Consciousness
- The value alignment problem is ill-defined because there is no universally accepted ethics or morals across cultures and religions, and even if we could perfectly program fuzzy ethical concepts into an AI and make it follow them, we still don't agree on what to program in.Roman Yampolskiy
- The multi-agent value alignment problem can be sidestepped by giving every individual their own personal virtual universe where they can do whatever they want, reducing the intractable problem of aligning with 8 billion disagreeing humans to the far easier problem of aligning with a single agent at a time.Roman Yampolskiy