causal

Off-policy data hurts when states are unreachable

Off-policy training in AlphaGo is fine—and even helpful in a DAgger sense—as long as the replay buffer is mostly states the current policy would visit plus a reasonable band of nearby states that funnel back to the optimal trajectory; it becomes harmful when MCTS relabels states the policy would never reach, since the network then wastes capacity learning good actions on unreachable states.

causalpending

Speaker

Eric Jang

Evidence Quote

“what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories.”

Source

Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:56 AM

Your notes

Loading notes...