causal
Off-policy data hurts when states are unreachable
Off-policy training in AlphaGo is fine—and even helpful in a DAgger sense—as long as the replay buffer is mostly states the current policy would visit plus a reasonable band of nearby states that funnel back to the optimal trajectory; it becomes harmful when MCTS relabels states the policy would never reach, since the network then wastes capacity learning good actions on unreachable states.
causalpending
Speaker
Eric JangEvidence Quote
“what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories.”
Created: 6/13/2026, 3:37:56 AM
Your notes
Loading notes...