RLHF
Reinforcement Learning from Human Feedback—a training method where humans choose between model-generated responses, and the model learns to align with their preferences over many comparisons.
As defined in cwt_brendan-foody
Where it comes up · 29
- Ai Sota #20266 claims
- Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history5 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough4 claims
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI4 claims
- Paul Christiano — Preventing an AI takeover4 claims
- Dario Amodei4 claims
- State-Of-The-Art Prompting For AI Agents3 claims
- Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality3 claims
and 21 more
Most important claims · 10 of 62Search all claims for “RLHF” →
- DeepSeek-R1-Zero showed that reasoning behaviors (backtracking, self-checking, 'wait let me verify') emerge naturally from large-scale reinforcement learning on verifiable questions with no human preference data or human-written reasoning chains added, analogous to AlphaZero removing the human prior to become more powerful.Deepseek Dylan Patel Nathan Lambert
- RLVR can be trained almost unlimitedly with continued benefit because you let the model solve more and more complex problems, whereas RLHF is preference tuning that reaches a point where more budget makes no sense—since people give different but equally correct answers, RLHF only learns an average preferred style that plateaus.Ai Sota #2026
- RLVR follows a scaling paradigm where log-increasing training compute yields a linear increase in evaluation scores (reproduced by OpenAI's o1 and DeepSeek), but there is no such scaling law for RLHF—in fact the seminal RLHF scaling paper is about reward model over-optimization—which is field-defining because to do the best RLVR you need the extra 10-100x compute but for RLHF you don't.Ai Sota #2026
- Pretraining on next-word prediction across trillions of tokens forces a model to store many concepts, and a second stage of RLHF on helpful, honest examples turns that base into an assistant, with the second stage using far less data and compute.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- There is a danger in alignment succeeding too well: whoever ends up in charge of AI systems—companies or governments—would gain fine-grained control over autonomous minds (e.g. turning up a 'loyalty feature'), raising both the ickiness of controlling an autonomous mind and the value lock-in risk, which is a strong reason to be extremely open and publish the constitutions models are expected to abide by.Sholto Douglas & Trenton Bricken — How LLMs actually think
- The ideal interpretability deployment gate for a highly capable model (e.g. GPT-7/ASL-4) is finding a robust, specific deception circuit—features across layers forming a circuit that lights up only when the model is being maliciously deceptive (not merely doing helpful theory-of-mind)—then ablating identified circuits and confirming under a battery of tests the behavior cannot be replicated, which Sholto argues is a far more precise safety tool than RLHF.Sholto Douglas & Trenton Bricken — How LLMs actually think
- There is a fundamental difference between raw LLM-generated data and LLM-generated data with a human in the loop who verifies it—even if that verification is a small percentage—because the human filtering and use of the LLM correctly is labor that produces higher-quality output, analogous to how labeled data is created for RLHF.Ai Sota #2026
- Yann LeCun was right in 2016 — controversially at the time when RL was hot — that RL is just the cherry on the cake, supervised learning is the icing, and the bulk of intelligence is in the unsupervised (self-supervised) cake; this turned out to be the recipe for ChatGPT, where most compute goes to next-token pre-training, then supervised fine-tuning, then RLHF for conversational ability.Aravind Srinivas
- Censoring or removing specific facts (like Tiananmen Square) from a language model is practically impossible because you'd have to filter them out of the pre-training data at every stage, and quality filters and word-play workarounds make complete removal infeasible; the internet's data also imparts a slight inherent left bias regardless of intent.Deepseek Dylan Patel Nathan Lambert
- OpenAI's Model Spec frames model behavior around roughly four stakeholder classes—the end user, the developer using the API, the platform (OpenAI, avoiding legal risk), and the rest of humanity—whose demands sometimes conflict; the rough heuristic is that models mostly follow user/developer instructions and be helpful, but block usage that harms others, staying neutral and non-paternalistic.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI