credit assignment
The problem of determining which of many actions in a trajectory were actually responsible for the eventual reward or outcome.
As defined in Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
Where it comes up · 2
- Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch3 claims
- The Brain’s Learning Algorithm Isn’t Backpropagation1 claim
Most important claims · 4 of 4Search all claims for “credit assignment” →
- MCTS acts as a policy-improvement operator: for every action taken in a game (whether the game was won or lost), the search provides a strictly better action/distribution to imitate, giving one low-variance supervision target per move rather than a single noisy win/loss signal for the whole trajectory.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- Naive win-reinforcement self-play RL is extremely high variance because in a matchup between evenly matched policies, only one critical move in one game out of ~100 (each ~300 moves) actually drove the win, so you have one useful supervision signal buried in ~99×300 neutral labels; the gradient variance grows quadratically with trajectory length T.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- The credit assignment problem is the fundamental challenge that computational systems must solve: determining which parameters to adjust and by how much to achieve a desired outputThe Brain’s Learning Algorithm Isn’t Backpropagation
- The reason MCTS is so much more sample-efficient than LLM/robotics RL is that you can locally improve your policy per move—the search, truncated by a working value function, suggests a better next move even without unrolling the whole trajectory or being able to win the game—whereas in normal LLM/robotics RL there is no way to evaluate and improve a move independent of actually solving the whole problem.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch