factual
LLM RL treats the whole sequence as one action
Current LLM RL treats an entire generated sequence as a single action with T=1, summing token log-probs into one log-prob, because decomposing into per-token rewards introduces cross-correlations between actions that magnify gradient variance—this is why LLMs do one-step rather than multi-step RL.
factualpending
Speaker
Eric JangEvidence Quote
“when you decompose the problem into a multi-step problem, you're now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance”
Created: 6/13/2026, 3:37:56 AM
Your notes
Loading notes...