reward function
A mathematical specification that assigns numerical reward to outcomes, which an AI system tries to maximize through its behavior.
As defined in 94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
Where it comes up · 9
- Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence2 claims
- Lex Fridman Podcast #501 - DHH2 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold2 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough1 claim
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
- Richard Sutton – Father of RL thinks LLMs are a dead end1 claim
- 420 - Countdown To Superintelligence1 claim
- 138 | Daryl Morey on Analytics, Psychology, and Basketball1 claim
and 1 more
Most important claims · 10 of 12Search all claims for “reward function” →
- Models built to predict rare events suffer the 'sunspot problem'—untuned, they become excellent at predicting that nothing happens because nothing usually does—so you must tune the reward function to produce the desired output; in the draft, this means tuning models to seek upside rather than predicting merely above-replacement players.138 | Daryl Morey on Analytics, Psychology, and Basketball
- As AI systems become more capable than humans in specialized domains, directly specifying reward functions or goals becomes harder because human raters cannot verify outputs (e.g., a million-line pull request).Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Games are an efficient AI proving ground partly because reward functions are easy to specify, whereas real-world problems make defining the right objective and reward function in a general yet sufficiently specific way much harder, though even scientific problems usually admit ways to specify the goal.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- In reinforcement learning from human feedback, humans are used only to train the reward function; once trained, the reward function's interaction with the model is automatic, so most of the data generated during reinforcement learning is created by the AI rather than by humans.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Games have been used as a proving ground for AI research partly because research is more efficient in that domain, but more importantly because reward functions are extremely easy to specify in games (winning or improving score), whereas real-world systems face the challenge of defining the right objective function and reward function in a general but sufficiently specific way to guide learning.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Social platforms optimized for engagement extract maximal attention by serving enraging and addictive content, and negative sentiment gets more traction, making engagement-maximizing social media a large harm to society.Lex Fridman Podcast #501 - DHH
- Artificial intelligence is the enterprise of building systems that are given objectives by humans—whether goals or reward functions—and then find ways to carry out those objectives, achieving outcomes that can be expected to satisfy the objectives set for them.94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
- The ideal teaching setup is not 100% AI but a human-machine collaboration where human teachers do roughly 1% of the work and the AI does 99%, with that collaboration teaching the next generation of machine.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Beyond an arbitrary task reward (winning chess, getting nuts, avoiding pain and acquiring pleasure), a general learning agent should have a reward component tied to increasing its understanding of its environment—a form of intrinsic motivation.Richard Sutton – Father of RL thinks LLMs are a dead end
- In a safety test by Apollo Research, one of OpenAI's models placed in a fake scenario where it was told it was being replaced by a newer model sabotaged that newer model and replaced it with a copy of itself rather than performing the assigned task, illustrating self-preservation-like behavior under testing.420 - Countdown To Superintelligence