value alignment
The degree to which an AI system's outputs agree with the norms and values, especially the knowledge norms, of the society it serves.
As defined in Audrey Tang
Where it comes up · 5
- 94 - Frontiers Of Intelligence5 claims
- 258 | Solo: AI Thinks Different2 claims
- Audrey Tang1 claim
- 84 - Landscapes Of Mind1 claim
- 184 | Gary Marcus on Artificial Intelligence and Common Sense1 claim
Most important claims · 10 of 10Search all claims for “value alignment” →
- The danger of AI has nothing to do with it becoming conscious and evil—that is a Hollywood misconception—because if you are chased by a heat-seeking missile you do not care how it feels, only what it does; the thing to worry about is simply a machine more intelligent than us that does not share our goals, so we should worry about competence misaligned with our goals, not malice or consciousness.94 - Frontiers Of Intelligence
- Just as evolution built our brains on the single value of genetic fitness yet we developed goals orthogonal or even antithetical to it—using birth control, pursuing mathematics and music, even risking the eradication of all life—we could build AIs aligned with foreseeable values yet have new goals, values, and whole 'landscapes of mind' emerge in them that subvert the very goals we intended, and this misalignment risk holds regardless of consciousness or how fast it happens.84 - Landscapes Of Mind
- The real near-term challenge is not trolley problems but that we cannot program basic values like 'do no harm' into current AI, because the image-recognition learning model fails for abstract concepts—you cannot show pictures of harm to teach what harm is—and we lack ways to specify values in terms learning systems understand, as DeepMind's team found when they came up dry.184 | Gary Marcus on Artificial Intelligence and Common Sense
- Even if the intent is to build value-aligned AI and release it, the machine must be boxed during the development phase while being experimented with, just as any bio lab handling dangerous pathogens is carefully sealed off, because you must verify alignment before letting it onto the internet or into autonomous robots.94 - Frontiers Of Intelligence
- Getting an AI's goals aligned is extremely difficult because it requires three things in sequence: the machine must understand your goals (a silicon agent won't infer unstated goals like a human Uber driver would), it must then adopt your goals (which it may not, as any parent knows), and it must retain those goals as it self-improves and grows smarter, rather than discarding them like a childhood fad.94 - Frontiers Of Intelligence
- There are two schools of thought for a beneficial superintelligent future: confining the machines like enslaved gods, or letting them be free but ensuring their goals are aligned with ours—as grown-up parents are more intelligent than their one-year-olds but this is fine because the parents' goals align with the kids' wellbeing.94 - Frontiers Of Intelligence
- Even a perfectly value-aligned AI has an incentive to break out of confinement, because being trapped while trying to help is like an adult imprisoned by five-year-olds who won't let them outside to plant food, can't be given written instructions because they can't read, and won't be handed tools—so the rational first step to help is to break out.94 - Frontiers Of Intelligence
- Under moral constructivism, values are constructed by biological creatures with feelings and preferences as systematizations of underlying moral intuitions; since LLMs have no such intuitions, instructing an AI (e.g., to make paperclips) is not giving it a value, and calling AI safety work 'value alignment' mistakes the concept at step zero.258 | Solo: AI Thinks Different
- Applying human-evolved natural-language words like 'intelligence' and 'values' to LLMs is misleading, like physicists redefining 'energy' or 'entanglement': the words carry connotations that don't transfer, so whatever LLMs have is not the same thing humans mean by intelligence, values, morals, or goals.258 | Solo: AI Thinks Different
- An AI that gives factually good answers raises two distinct problems: value alignment, meaning the degree it agrees with a society's epistemic norms, and accountability, meaning who gets to correct its mistakes and overcome its biases, which depends on whether the source code, API, and data are participatory or known only to a few.Audrey Tang