value function
In reinforcement learning, a function estimating the expected future reward of a state or action, allowing credit assignment before a final outcome.
As defined in Ilya Sutskever – We're moving from the age of scaling to the age of research
Where it comes up · 10
- Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch7 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end4 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research4 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research4 claims
- Rich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 20254 claims
- Adam Marblestone – AI is missing something fundamental about the brain2 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end2 claims
- Ilya Sutskever: Deep Learning | Lex Fridman Podcast #941 claim
and 2 more
Most important claims · 10 of 30Search all claims for “value function” →
- Efficiency does not exist in a vacuum and only makes sense when yoked to a goal: a state efficient at building dirty energy and one efficient at clean energy are opposites, so any 'efficiency' or AI deployment must be evaluated by the value function it optimizes—and DOGE's revealed goal is to serve Donald Trump's movement, not any articulated public outcome.Derek Thompson
- Temporal difference learning solves the sparse-long-term-reward problem (e.g. a 10-year startup goal or winning chess) via a value function that predicts the long-term outcome; intermediate progress (taking a piece, hitting a milestone) raises the value prediction, and that increase immediately reinforces the action that produced it.Richard Sutton – Father of RL thinks LLMs are a dead end
- Optimal decision making at its core, regardless of method, requires considering counterfactuals—asking whether doing one thing instead of another would be better—whether answered via a learned simulator, a value function, or a reward model; thus the key is not great simulation but a mechanism for answering counterfactuals.Fully autonomous robots are much closer than you think – Sergey Levine
- A value function lets a network look at a board and instantly predict win probability—amortizing a huge number of possible game playouts into a single forward pass—just as humans glance at a board and 'know' they will lose without playing it out; this is the key to truncating search depth in Go.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- The base common model of the agent has four parts: a policy (what to do in the current situation), a value function (learned via TD learning, producing a number for how well things are going used to adjust the policy), a perception/state-representation component, and a transition model of the world (belief about consequences of actions, learned richly from all sensation, not just reward).Richard Sutton – Father of RL thinks LLMs are a dead end
- MCTS is only guaranteed to improve the policy if the value estimates at the leaves are accurate; if training data lacks late-stage resolutions (e.g., bots resign rather than play to Tromp-Taylor end), bad terminal values propagate up through backups and PUCT, making MCTS produce a worse distribution than the raw policy—so the value function must be grounded, e.g. by forcing some games to resolve fully.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- A basic model of an agent has four parts: a policy that says what to do in a situation, a value function that is learned with TD learning and produces a number about how well it's going, a perception component that constructs state representation, and a transition model of the world that represents what will happen if you take an action.Richard Sutton – Father of RL thinks LLMs are a dead end
- Aligning AGI to human values is a question about training a value function (objective/goal) that is learned separately from the agent's main capability function, similar to how humans have internal rather than external reward functions that they can choose to change.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
- Late-stage Go positions are nearly decidable—with most stones placed there is little tree-depth uncertainty—so games played to the end by reasonable players give good training data for the value function, and the hardest part to learn is who is winning in the mid-game.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- Temporal difference (TD) learning solves the sparse reward problem by learning a value function that predicts long-term outcomes; when intermediate progress is made (like capturing an opponent's piece in chess), the value prediction increases immediately, creating intermediate reinforcement without explicit reward.Richard Sutton – Father of RL thinks LLMs are a dead end