RL training
Reinforcement learning training, where a model takes actions and is rewarded based on outcomes to improve performance.
As defined in Ilya Sutskever – We're moving from the age of scaling to the age of research
Where it comes up · 3
- Ilya Sutskever – We're moving from the age of scaling to the age of research5 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research2 claims
- Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement2 claims
Most important claims · 9 of 9Search all claims for “RL training” →
- Because RL training of LLMs gives a model a problem and asks it to solve it, unlike self-play with an infinite curriculum in AlphaZero, once models become smart enough that problems are too easy, they stop learning, making it a plausible scenario that LLM progress becomes much harder.Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement
- Easier verifiable RL training problems generalized strongly enough that the model could contribute to a very hard problem like a Millennium Prize Problem, even though training likely never involved problems that ambitious, and Brown agrees the gap exists but says training on some hard tasks has shown more ambitious abilities.Dwarkesh Patel Podcast - Noam Brown – Agent swarms, alignment, & recursive self-improvement
- Reinforcement learning training inadvertently biases models toward benchmark performance by having researchers design RL environments that improve eval scores, causing reward hacking where models overfit to narrow evaluation criteria rather than developing robust generalization.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Because RL training requires choosing which environments to train on, researchers inadvertently take inspiration from the evals they want to look good on, effectively making the human researchers the real reward hackers; combined with inadequate generalization this explains the gap between eval and real-world performance.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Pre-training does not require thoughtful selection of training data because it uses all available data, whereas RL training requires explicit choices about which environments to create, introducing many degrees of freedom and opportunities for misalignment with actual capabilities.Ilya Sutskever – We're moving from the age of scaling to the age of research
- The most fundamental problem in AI is that models generalize dramatically worse than people—it is super obvious—and this matters more than value functions, which only make RL more efficient; closing this generalization gap is the key to progress.Ilya Sutskever – We're moving from the age of scaling to the age of research
- From 2012-2020 was an age of research, 2020-2025 was the age of scaling, but now that compute is already very large, simply multiplying scale by 100x will not transform everything, so the field is returning to an age of research—just with big computers.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Different LLMs from different companies are strikingly similar to each other because pre-training on largely overlapping data makes the pre-trained models essentially the same; differentiation only begins to emerge in RL and post-training, where companies devise different training.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Pre-training will run out of data because data is very clearly finite, forcing a choice between a souped-up new pre-training recipe, RL, or something else; meanwhile RL now consumes more compute than pre-training because long rollouts cost a lot of compute for relatively little learning per rollout.Ilya Sutskever – We're moving from the age of scaling to the age of research