factual
Scaling has more dimensions than compute and data
Scaling curves have more dimensions than the original compute, parameters, and data — notably inference compute and context length — and for long context with cheap inference, training an SSM may be worth a 10x worse training-scaling property because the inference budget for long context windows matters most.
factualpending
Speaker
Aman SangerEvidence Quote
“maybe the thing you want to train, is some kind of SSM. Because they’re much, much cheaper and faster at super, super long context. And even if, maybe it was 10 X more scaling properties during training... it’s worth it.”
Created: 6/13/2026, 3:31:02 AM
Your notes
Loading notes...