factual
RL has converged toward on-policy for stability
Modern RL has largely converged on on-policy setups that don't directly train on off-policy data—at best using off-policy data to reduce variance or shape rewards via advantage computation—because doing so is more stable and prevents off-policy dynamics problems from blowing up the loss.
factualpending
Speaker
Eric JangEvidence Quote
“It's just more stable. You might use the off-policy Q as a way to do advantage computation... If there's a problem in these dynamics, it doesn't blow up your loss as much.”
Created: 6/13/2026, 3:37:56 AM
Your notes
Loading notes...