RL
Reinforcement learning; training method where models learn by receiving rewards for actions, used to elicit reasoning and agency in language models.
As defined in AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
Where it comes up · 16
- Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"3 claims
- Dwarkesh Patel Podcast - Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 20283 claims
- Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary3 claims
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI3 claims
- Dwarkesh Patel Podcast - The data black hole at the center of AI3 claims
- Yann LeCun Says LLMs Have 2 Years Left…2 claims
- AGI is still 30 years away — Ege Erdil & Tamay Besiroglu2 claims
- AMA: career advice given AGI, how I research ft. Sholto & Trenton2 claims
and 8 more
Most important claims · 10 of 32Search all claims for “RL” →
- Training only on human-written data, even high quality, is hard for a model to fit perfectly, so reproducing ChatGPT would require iterative methods more like RL—such as iterative supervised fine-tuning where humans edit model-generated outputs—rather than a single round of fine-tuning on human data.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Equalizing pre-training, RL, and inference compute implies the number of inference tokens roughly equals the number of pre-training tokens; estimating ~50M tokens/sec for two months gives ~200 trillion inference tokens against a Chinchilla-optimal ~2 trillion for a ~100B-active-parameter model, suggesting frontier models are roughly 100x over-trained relative to Chinchilla-optimal.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- For RL-based training to work, the model must already have some nonzero prior probability of producing the correct solution, which is why mind-stretching amounts of human expert trajectories are needed in every skill the model should master.Dwarkesh Patel Podcast - The data black hole at the center of AI
- Capabilities like reasoning look cheap and easy in retrospect (only ~1% of compute on RL) only because we stand on a huge stack of technology built over years; in 2015 building a reasoning model would have looked insurmountable, requiring internet-scale data, tens of thousands of GPUs, and distillation innovations — so agency may similarly look simple in hindsight while taking years of complementary innovation to actually achieve.AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
- RL training is less efficient per token than pre-training for two reasons: you may not train a backward pass on all rollouts (so FLOPs per token can be as low as 2 rather than the full forward+backward 6), and RL involves substantial decode which often runs at lower MFU than training, together giving roughly a 30% efficiency penalty.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The main way AIs have improved is by adding more and better data and scaling the compute used to generate that data, with RL being the dominant method, which works as a kind of synthetic data generation against a verifier.Dwarkesh Patel Podcast - The data black hole at the center of AI
- Reinforcement learning can be understood as synthetic data generation: a large amount of compute is applied against a verifier to find good outputs, and the model is then trained to predict those correct rollouts much as it predicts next words in internet text.Dwarkesh Patel Podcast - The data black hole at the center of AI
- Although RL is in principle better than imitation learning, current models use imitation/supervised learning first because effective learning from one's own experience requires already knowing something about the task; supervised learning builds the prior-knowledge foundation that lets the model later figure things out quickly via RL, mirroring how LLMs go from next-token prediction to RL.Fully autonomous robots are much closer than you think – Sergey Levine
- RL training exhibits the same scaling behavior as pre-training: how well a model does on tasks (e.g., math contests like AIME, but also a wide variety of RL tasks including code) is log-linear in how long it is trained, mirroring pre-training scaling laws.Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"
- Near-term capability gains will come from training models to carry out longer, more involved projects (e.g. an entire coding project: writing files, testing, iterating) rather than single steps, and because this training area is new there is a lot of low-hanging fruit in it.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI