causal

Larger context windows trade off instruction following

Increasing context window size has a trade-off with instruction-following capability: vendors advertise needle-in-a-haystack metrics but say less about whether instruction-following degrades, and throwing more information at a model can make it more confused (more entropy to handle) rather than more capable — though large contexts could greatly improve internal/personal-file search, which web-style indexing has never cracked.

causalpending

Speaker

Aravind Srinivas

Evidence Quote

“most people, when they advertise new context window increase, they talk a lot about finding the needle in the haystack sort of evaluation metrics and less about whether there’s any degradation in the instruction following performance.”

Source

Aravind Srinivas— Lex Fridman Podcast
Created: 6/13/2026, 3:26:05 AM

Your notes

Loading notes...