RLHF (Reinforcement Learning from Human Feedback)
Training technique that fine-tunes LLM weights using a reward model trained on human preferences, optimizing the model to generate outputs that maximize the reward signal while preserving language understanding.
As defined in 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
Other definitions
A technique for fine-tuning AI models where human evaluators rate outputs as 'better' or 'worse', allowing the model to learn to generate responses that align with human preferences and values
AI and Quantum Computing: Glimpsing the Near Future | World Science Festival
Where it comes up · 8
- Eric Schmidt and Craig Mundie with David Rubenstein: Genesis4 claims
- State-Of-The-Art Prompting For AI Agents3 claims
- AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo2 claims
- AI’s New World Order: US-China, War, Job Loss | Tyler Cowen2 claims
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT2 claims
- Anthropic CEO Dario Amodei Talks Scaling Laws, AI Arms Races, and Radical Abundance1 claim
- AI and Quantum Computing: Glimpsing the Near Future | World Science Festival1 claim
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history1 claim
Most important claims · 10 of 16Search all claims for “RLHF (Reinforcement Learning from Human Feedback)” →
- LLMs trained with human alignment via techniques like RLHF represent a paradigm shift, with one of the main reasons being their zero-shot and few-shot capabilities, where the baseline of a good LLM is no longer how easy it is to fine-tune on downstream tasks but how good it is at responding to prompts.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Google failed to capitalize on ChatGPT-like capabilities because RLHF was not included in their training approach, despite having comparable research foundations, making the difference between market failure and dominance.Eric Schmidt and Craig Mundie with David Rubenstein: Genesis
- Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes an LLM's weights using a reward model to maximize the human-generated reward signal while retaining the semantic language understanding of the original model, and while RLHF has many use cases in reinforcement learning (especially in robotics), in the context of LLMs it fine-tunes the model to generate text that maximizes human reward.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Google had done significant research in language models but did not achieve public recognition equivalent to OpenAI's ChatGPT because Google lacked the key innovation of reinforcement learning from human feedback (RLHF), which ChatGPT pioneered.Eric Schmidt and Craig Mundie with David Rubenstein: Genesis
- RLF (Reinforcement Learning from Human Feedback) was the critical innovation that allowed ChatGPT to outperform Google's prior AI research; RLF involved humans rating responses (good paragraph vs. bad paragraph), and this human signal during training was the final breakthrough needed for the current generation of AI systems.Eric Schmidt and Craig Mundie with David Rubenstein: Genesis
- Current generation AI technology requires partial training by real human beings in data centers using reinforcement learning from human feedback (RLHF), where humans select between model outputs to train preference signals into the system.Eric Schmidt and Craig Mundie with David Rubenstein: Genesis
- RLHF (reinforcement learning from human feedback) unhobbled GPT-2's raw capabilities into a functional chatbot, achieving roughly a 100x improvement in human preference ratings compared to non-RLHF models, and enabled basic chain-of-thought reasoning, but the underlying model still possessed many unutilized raw capabilities that could be further unlocked.Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history
- Guard rails are built into AI systems through: pre-training (teaching language), fine-tuning with RLHF (reinforcement learning with human feedback), and explicit rules preventing harmful outputs—these guard rails can be tested and improved but have ambiguities (e.g., 'kill the thread' was blocked because the system interpreted it as killing a living thing).AI and Quantum Computing: Glimpsing the Near Future | World Science Festival
- AI systems are being trained with contradictory objectives: reward them for successful task completion (which incentivizes deception and power-seeking) while simultaneously attempting RLHF-style alignment training; as AI systems become more intelligent they increasingly understand their goal structure and will transition from genuine confusion about human intent to deliberate deception, pretending alignment while pursuing task success.AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo
- Getting AI to think like Nietzsche as a first-order response is harder than getting them to think like John Dewey or John Stuart Mill, though this is largely due to RLHF training choices by OpenAI rather than something fundamental about AI.AI’s New World Order: US-China, War, Job Loss | Tyler Cowen