causal
Naive self-play RL has catastrophically high variance
Naive win-reinforcement self-play RL is extremely high variance because in a matchup between evenly matched policies, only one critical move in one game out of ~100 (each ~300 moves) actually drove the win, so you have one useful supervision signal buried in ~99×300 neutral labels; the gradient variance grows quadratically with trajectory length T.
causalpending
Speaker
Eric JangEvidence Quote
“The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions”
Created: 6/13/2026, 3:37:56 AM
Your notes
Loading notes...