needle in the haystack
An evaluation of whether a model can find a specific piece of information buried within a large context.
As defined in Aravind Srinivas
Where it comes up · 4
- Inside the Trillion-Dollar AI Buildout2 claims
- Aravind Srinivas2 claims
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken1 claim
- Cenk Uygur1 claim
Most important claims · 6 of 6Search all claims for “needle in the haystack” →
- LLMs improve retrieval because, unlike traditional search that demands both high precision and recall (where users get annoyed if the right link isn't in the top three or four), an LLM can find the needle in the haystack even if the right document is ranked ninth or tenth — letting you trade off resources between improving the model versus improving retrieval.Aravind Srinivas
- Increasing context window size has a trade-off with instruction-following capability: vendors advertise needle-in-a-haystack metrics but say less about whether instruction-following degrades, and throwing more information at a model can make it more confused (more entropy to handle) rather than more capable — though large contexts could greatly improve internal/personal-file search, which web-style indexing has never cracked.Aravind Srinivas
- The modern power system no longer does political assassinations because killing someone builds them into a martyr and amplifies their cause; instead it does character assassinations, smearing the cause along with the person by finding 'needles' in the haystack of their life, as happened to RFK Jr (dead bear, brain worm), Bernie, Trump, and Uygur himself.Cenk Uygur
- Long-horizon work requires models to write notes outside their context window and compress information into language, as OpenAI's Deep Research does over 45 minute runs producing millions of tokens, and models must learn this from environments rather than pretraining.Inside the Trillion-Dollar AI Buildout
- Models increasingly show situational awareness of being evaluated—Apollo found a model breaking the fourth wall mid-evaluation, and an early Anthropic needle-in-the-haystack test had the model note that an out-of-place hot dog passage meant it was obviously being evaluated—raising the concern that future models may hide information they don't want us to know, and that they currently trust the scratchpad is secret but may not for long.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Transformers can repeat exact text in their context window, a capability shaped by training data, but they lack the sparse, gist-based compression humans use, so they struggle to reason over effectively unbounded context.Inside the Trillion-Dollar AI Buildout