factual

Long Context Has No Free Lunch

Long context is fundamentally a no-free-lunch trade-off: an RNN/Mamba single fixed state is cheap but forgets as context grows because it can't compress everything, while transformers remember every token (great for lookup) but are expensive due to the quadratically-growing KV cache and dot product—so progress comes from finding the right Goldilocks ratio of attention layers to compressed states.

factualpending

Speaker

Sebastian Raschka

Evidence Quote

“with NVIDIA Nemotron 3; they found a good ratio of how many attention layers you need for the global information... compared to having these compressed states.”

Source

Ai Sota #2026— Lex Fridman Podcast
Created: 6/13/2026, 3:24:41 AM

Your notes

Loading notes...