transition model
A learned model of how the environment transitions between states given actions, used to extract maximum signal from observations.
As defined in Some thoughts on the Sutton interview
Where it comes up · 4
- Rich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 20255 claims
- Some thoughts on the Sutton interview2 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end1 claim
- Richard Sutton – Father of RL thinks LLMs are a dead end1 claim
Most important claims · 9 of 9Search all claims for “transition model” →
- A basic model of an agent has four parts: a policy that says what to do in a situation, a value function that is learned with TD learning and produces a number about how well it's going, a perception component that constructs state representation, and a transition model of the world that represents what will happen if you take an action.Richard Sutton – Father of RL thinks LLMs are a dead end
- An LLM being RLed on outcome-based rewards learns only O(1) bits per episode even though the episode may be tens of thousands of tokens long, whereas animals extract far more information from interacting with their environment than just the end-of-episode reward signal.Some thoughts on the Sutton interview
- An option is a pair consisting of a policy (way of behaving) and a termination function (decision to stop behaving), and the agent learns what happens when it follows each option to form a high-level transition model that enables planning with larger temporal jumpsRich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
- Anything that can be learned can also be planned—general value functions used to learn models and values can equally be used to plan, because planning just applies value iteration to a different set of states, meaning there is mathematical unity across learning and planningRich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
- The big world perspective states that the world is much larger and more complex than the agent—containing billions of other agents and countless atomic details—making it impossible for the agent to hold complete world knowledge, thereby necessitating approximate value functions, policies, and transition modelsRich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
- In the OAK architecture, the agent learns a high-level transition model of the world by understanding what happens when following different options, enabling planning with larger temporal jumps and carving the world at its natural jointsRich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
- The base common model of an agent has four parts: a policy (what to do in the current situation), a value function (a learned number indicating how well things are going, used to adjust the policy), a perception component (constructing the state representation), and a transition model of the world (predicting the consequences of actions), the last of which is learned richly from all sensation rather than from reward.Richard Sutton – Father of RL thinks LLMs are a dead end
- The OAK architecture contains eight main steps executed in parallel at runtime: learning reward policy and value, generating new state features, creating subproblems from features, solving subproblems to create options, learning transition models for options, ordering features by utility, maintaining metadata and pruning, and planning with the learned modelsRich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
- If you tried to fit the transition-model spec into LLMs by fine-tuning on all observed tokens, in practice the most naive way of doing this doesn't work well, according to researcher friends.Some thoughts on the Sutton interview