factual
Long Context Has No Free Lunch
Long context is fundamentally a no-free-lunch trade-off: an RNN/Mamba single fixed state is cheap but forgets as context grows because it can't compress everything, while transformers remember every token (great for lookup) but are expensive due to the quadratically-growing KV cache and dot product—so progress comes from finding the right Goldilocks ratio of attention layers to compressed states.
factualpending
Speaker
Sebastian RaschkaEvidence Quote
“with NVIDIA Nemotron 3; they found a good ratio of how many attention layers you need for the global information... compared to having these compressed states.”
Created: 6/13/2026, 3:24:41 AM
Your notes
Loading notes...