RL from Verifiable Rewards
Reinforcement learning where the reward comes from an objectively checkable signal (like a correct math answer or passing unit tests) rather than subjective human preference.
As defined in Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
Where it comes up · 1
Most important claims · 7 of 7Search all claims for “RL from Verifiable Rewards” →
- Rates of AI research progress almost don't change much so long as the model can completely implement ideas, because even getting a Noam Shazeer at 100x speed—or accelerating an imperfect version of his intuition by 100x—is wild; ML research is also one of the easier things to RL on since the objective (did the loss go down) is well-defined.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- RL is not merely narrowing the base model's existing distribution; provided the reward signal is sufficiently clean, RL can impart genuinely new knowledge and superhuman capabilities to a neural net (as DeepMind's Go and chess agents demonstrated), and the Tsinghua paper suggesting otherwise used far less RL compute than the base model, so it does not establish a structural ceiling.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Current RL jaggedness mirrors the GPT-2 era when fine-tuned models excelled only at their fine-tuned tasks; just as scaling pre-training compute led GPT-4 to generalize across sub-tasks better than smaller fine-tuned models, scaling total RL compute will produce the same transition to unsupervised meta-learning and broad generalization across reasoning domains—the interp agent generalizing without ever being RL'd for interp is early evidence.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- RL from verifiable rewards works by using clean, true reward signals (like correct math answers or passing unit tests) rather than the noisy pairwise human feedback of RLHF, because humans are poor judges with biases like length bias; however even verifiable signals like unit tests can be hacked by models hard-coding values when they discover the actual test.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- As labs expand RL from unit tests to long-horizon goals like 'make money on the internet', such broad goals with clear objective functions are good RL tasks at high capability but carry incredible scope for misalignment.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- The biggest change in the last year is that RL in language models has finally worked: there is now proof of an algorithm that delivers expert human reliability and performance given the right feedback loop, conclusively demonstrated so far in competitive programming and math.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Counterintuitively, longer-horizon tasks can be easier to judge if you can ever get the reward—'make money on the internet' is an easy reward signal to evaluate—so the key is being able to pre-train up to easy-to-judge end signals (does the website work, do people like it); if you could pre-train on every screen in America, the achievable RL tasks differ radically from those possible using only today's internet.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken