factual

Q-learning plans over visited trajectories, MCTS over unvisited

There is a deep duality between MCTS and Q-learning: both propagate value information through a recursive dynamic-programming structure, but MCTS plans over trajectories the agent hasn't visited yet (forward search), while Q-learning plans over trajectories it has already visited—Q-learning mattered historically because we couldn't do forward search on high-dimensional problems like robotics.

factualpending

Speaker

Eric Jang

Evidence Quote

“Q-learning, or approximate dynamic programming, propagates what you know about the future Qs backward like this. You can see that there's a similar structure going on here”

Source

Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:56 AM

Your notes

Loading notes...