long context
Models processing very long input/output sequences, which enlarges the KV cache and drives memory demand.
As defined in Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute
Where it comes up · 3
- Is RAG Still Needed? Choosing the Best Approach for LLMs10 claims
- Inside the Trillion-Dollar AI Buildout1 claim
- Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute1 claim
Most important claims · 10 of 12Search all claims for “long context” →
- Prompt caching can partially offset long context compute inefficiency for static data, but for dynamic data streams where content changes frequently, the full computational tax must be paid on every request.Is RAG Still Needed? Choosing the Best Approach for LLMs
- Long context creates a massive compute inefficiency: for a 500-page manual (~250k tokens), every user query requires reprocessing the entire manual. RAG, by contrast, pays the embedding cost only once at indexing time, not on every request.Is RAG Still Needed? Choosing the Best Approach for LLMs
- Long context is a brute-force approach where documents are put directly into the context window and the model's attention mechanism does the heavy lifting of finding the answer, rather than using a retrieval system.Is RAG Still Needed? Choosing the Best Approach for LLMs
- RAG is not dead; while long context wins on simplicity, RAG still has legitimate use cases and remains necessary in certain scenarios.Is RAG Still Needed? Choosing the Best Approach for LLMs
- Context injection—the problem of getting the right data into an LLM at the right time—has two fundamentally different solution approaches: RAG (retrieval augmented generation), which is an engineering approach, and long context, which is a model-native solution.Is RAG Still Needed? Choosing the Best Approach for LLMs
- Long context solves the whole book problem by dumping the full documents (e.g., the complete requirements document and complete release notes) into the context window, allowing the model to perform the full comparison needed to identify omissions.Is RAG Still Needed? Choosing the Best Approach for LLMs
- With long context there is no retrieval step, so the model gets to see everything, eliminating the silent failure problem.Is RAG Still Needed? Choosing the Best Approach for LLMs
- The memory crunch's near-term constraint is not EUV but that vendors didn't build fabs for years—they lost money on memory in 2023—so even after prices reflected reasoning/KV-cache demand (a year lag) and vendors began building (another 3-6 months), the two-year fab construction time means no meaningful new memory fabs come online until late 2027 or 2028.Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute
- For problems involving a bounded dataset that requires complex global reasoning (such as analyzing a specific legal contract or summarizing a book), long context is the way to go because it simplifies the stack and improves reasoning.Is RAG Still Needed? Choosing the Best Approach for LLMs
- If we accept that we can fit whatever data we need into the context window, the primary argument for choosing long context over RAG reduces to a single factor: simplicity.Is RAG Still Needed? Choosing the Best Approach for LLMs