factual

RL has converged toward on-policy for stability

Modern RL has largely converged on on-policy setups that don't directly train on off-policy data—at best using off-policy data to reduce variance or shape rewards via advantage computation—because doing so is more stable and prevents off-policy dynamics problems from blowing up the loss.

factualpending

Speaker

Eric Jang

Evidence Quote

“It's just more stable. You might use the off-policy Q as a way to do advantage computation... If there's a problem in these dynamics, it doesn't blow up your loss as much.”

Source

Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:56 AM

Your notes

Loading notes...