imitation learning
Learning to reproduce behavior or outputs from observed data (e.g., predicting the next human-written token), as opposed to learning from self-generated experience and reward.
As defined in Some thoughts on the Sutton interview
Where it comes up · 4
- Some thoughts on the Sutton interview4 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end2 claims
- Applied Intuition: A Billion Intelligent Machines1 claim
- Fully autonomous robots are much closer than you think – Sergey Levine1 claim
Most important claims · 8 of 8Search all claims for “imitation learning” →
- Humanity's success rests on accumulating knowledge over tens of thousands of years, inheriting language, legal systems, and most technological knowledge from prior generations, a process more analogous to imitation learning than to RL from scratch.Some thoughts on the Sutton interview
- Imitation learning is not categorically different from RL; it is just short-horizon RL where the episode is a single token long: the LLM conjectures the next token from its world understanding and receives reward proportional to how well it predicted the true token.Some thoughts on the Sutton interview
- Patel's main disagreement with Sutton is that the concepts Sutton uses to distinguish LLMs from true intelligence are not actually mutually exclusive and dichotomous: imitation learning is continuous with and complementary to RL, and models of humans can provide a prior that facilitates learning true world models.Some thoughts on the Sutton interview
- End-to-end imitation learning alone does not yield productionizable autonomy, but combining imitation learning with reinforcement learning inside a high-performance simulation smooths out problem cases and is the technical path to large-scale, widely deployed physical AI.Applied Intuition: A Billion Intelligent Machines
- Although RL is in principle better than imitation learning, current models use imitation/supervised learning first because effective learning from one's own experience requires already knowing something about the task; supervised learning builds the prior-knowledge foundation that lets the model later figure things out quickly via RL, mirroring how LLMs go from next-token prediction to RL.Fully autonomous robots are much closer than you think – Sergey Levine
- Evolution does meta-RL to make an RL agent that can selectively do imitation learning, whereas with LLMs we go the opposite way: first building a base model that does pure imitation learning, then doing RL on it to make it a coherent agent with goals and self-awareness.Some thoughts on the Sutton interview
- Cultural knowledge (like complex multi-step skills such as hunting seals) that cannot be reasoned through must be transmitted across generations via imitation, but this imitation is a small thing layered on top of the basic trial-and-error and prediction learning that we share with animals.Richard Sutton – Father of RL thinks LLMs are a dead end
- Infants do not learn primarily by imitation; they learn by trying things—waving hands, moving eyes, making sounds—for which there are no targets or examples, indicating their behavior is generated by trial-and-error rather than copying.Richard Sutton – Father of RL thinks LLMs are a dead end