causal

Naive self-play RL has catastrophically high variance

Naive win-reinforcement self-play RL is extremely high variance because in a matchup between evenly matched policies, only one critical move in one game out of ~100 (each ~300 moves) actually drove the win, so you have one useful supervision signal buried in ~99×300 neutral labels; the gradient variance grows quadratically with trajectory length T.

causalpending

Speaker

Eric Jang

Evidence Quote

“The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions”

Source

Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:56 AM

Your notes

Loading notes...