reinforcement learning from human feedback
A training method that tunes an AI model to produce outputs humans rate favorably, rather than outputs optimized for truth.
As defined in Alison Gopnik
Where it comes up · 3
- The Future of Artificial Intelligence3 claims
- Alison Gopnik1 claim
- 308 | Alison Gopnik on Children, AI, and Modes of Thinking1 claim
Most important claims · 5 of 5Search all claims for “reinforcement learning from human feedback” →
- Jailbreaking attempts get around reinforcement learning from human feedback safety training by using roleplay prompts, like asking ChatGPT to roleplay as a deceased grandmother who was a chemical engineer at a Napalm plant and would tell stories about Napalm production—leading the system to provide dangerous information it is supposedly trained not to provide.The Future of Artificial Intelligence
- The reason AI systems are effective is not the algorithms (gradient descent and transformers) but the data: hundreds of thousands of humans who put text and images on the web, plus reinforcement learning from human feedback and prompt engineering, supply the human exploratory and creative intelligence; tech companies then credit the algorithms much like the 'Stone Soup' folktale, in which the magic stone gets credit for soup actually made from villagers' contributions.308 | Alison Gopnik on Children, AI, and Modes of Thinking
- Large language models like ChatGPT become chatbots through a process of reinforcement learning from human feedback: humans rate different outputs for the same prompt, and the model is then trained to prefer the outputs that humans rated as better.The Future of Artificial Intelligence
- The three-part model of how generative AI works includes: (1) a monster from pre-training on unfiltered internet text (like an H.P. Lovecraft creature), (2) a human mask of supervised fine-tuning with human feedback, and (3) reinforcement learning from human feedback creating a happy face facade, suggesting the model's helpfulness is a thin layer over an alien intelligence underneath.The Future of Artificial Intelligence
- Reasoning systems still hallucinate because their objective function is not truth—unlike even a baby—but rather producing output a human will like, via reinforcement learning from human feedback; relatedly they reproduce reasoning patterns yet fail at basic arithmetic and at generating the novel relationships humans generate.Alison Gopnik