What this covers

This IBM Technology video, presented by Martin Keen according to the publisher's description, treats context injection (getting the right data to an LLM at the right time) as a choice between two designs. RAG is the engineering answer, and long context is the model-native one. The setup is historical. Early models had context windows of about 4K tokens, too small for whole documents, so retrieval was unavoidable. With windows now at a million tokens or more, roughly 700,000 words, the video asks whether the embedding and storage layers still earn their place.

The case for long context rests on simplicity. Dropping the database, embeddings, and retrieval logic yields what the video calls The No Stack Stack. It also removes Silent Failure, where the answer exists in the store but retrieval never surfaces it. It handles The Whole Book Problem too: spotting omissions, such as gaps between a requirements document and release notes, requires seeing both documents in full, and RAG can only retrieve what is present.

The video then turns to long context's weaknesses. Attention gets diluted at large sizes, so details buried mid-document can be missed or hallucinated, which is the Needle in Haystack problem. RAG sidesteps this by handing the model only the top few chunks. Cost is the other argument: a 500-page manual of about 250k tokens must be reprocessed on every query, while RAG pays its embedding cost once at indexing. Prompt caching helps for static data but not for changing streams.

The conclusion is a split by data shape. Bounded datasets needing global reasoning, such as a single contract or a book, suit long context. Enterprise knowledge measured in terabytes or petabytes still needs a vector database and a retrieval layer. The video describes RAG's production burden plainly: chunking strategy, embeddings, reranking, and keeping vectors synced with source data.

Sharpest takeaway

Long context windows and RAG represent competing architectural approaches to context injection for LLMs, each with distinct tradeoffs: long context wins on simplicity and reasoning for bounded datasets, while RAG remains necessary for infinite enterprise datasets despite higher infrastructure overhead.

  • Modern LLMs have context windows (1M+ tokens) large enough to fit entire documents, making the brute-force approach of direct context injection viable
  • Long context eliminates retrieval failure risk ('silent failure') but suffers from needle-in-haystack attention dilution at scale
  • RAG's infrastructure complexity (chunking, embeddings, vector databases, rerankers) is justified only when data exceeds context window capacity or retrieval efficiency is critical

The claims · ranked24 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.81

There is an intuitive assumption that if data is in the context window, the model will use it, but research suggests otherwise: as context windows grow to 500,000 tokens, the model's attention mechanism becomes diluted, and the model often fails to retrieve or hallucinates details from text that is buried in the middle of very large documents.

causalhigh valueestablishednovelty 3/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“There's a an intuitive assumption that if data is in the context window, the model's probably going to use it, but research suggests otherwise. Because as we start with a context window and then it grows and it continues to grow and now we're at like 500,000 tokens, well, the model's attention mechanism can get a bit diluted. If you ask a specific question about a single paragraph that's buried in, let's say, the middle of a 2,000 page document, well, the model often fails to retrieve it or it hallucinates details from the surrounding text.”

0.76

RAG introduces a critical failure point called 'silent failure': the answer exists in the database, but the LLM never sees it because the retrieval step fails to return the right results due to the probabilistic nature of semantic search.

causalhigh valueestablishednovelty 2/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“RAG introduces a critical point of failure here, the retrieval step itself, because when a user asks a question, RAG looks at mathematical representations of the data, which are stored in vectors. And vectors are basically just like a really long series of numbers in an array. And it tries to find the closest match. That's semantic search. But semantic search is probabilistic and for all manner of reasons, the retrieval might fail to find the relevant document. And we actually have a name for this. It's called silent failure. The answer, well, it existed in the data, but the LLM never saw it because the retrieval step didn't return the right results.”

0.74

RAG addresses the needle-in-haystack problem by retrieving only the top five most relevant chunks, removing the haystack and presenting the model with just the needles, forcing the model to focus on signal rather than noise.

causalhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“But with RAG, we're giving the model less noise. So by retrieving, say only the top five relevant chunks, RAG has removed the haystack and presents the model with just the needles. It forces the model to focus on the signal and not the noise.”

0.71

Long context creates a massive compute inefficiency: for a 500-page manual (~250k tokens), every user query requires reprocessing the entire manual. RAG, by contrast, pays the embedding cost only once at indexing time, not on every request.

causalhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Long context creates a massive compute inefficiency. So, if we take a manual, let's say this is like a a 500 page manual, and we've got to turn this into tokens. Well, that's something like 250k of tokens. And we need to do that every time we make a user query and we put this document in the prompt. You're basically requiring the model to process that manual every time. Now, RAG also has to process that manual, but it only pays that processing cost once at indexing time.”

0.71

Prompt caching can partially offset long context compute inefficiency for static data, but for dynamic data streams where content changes frequently, the full computational tax must be paid on every request.

causalhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Now, prompt caching that can partially offset some of this for static data, but for dynamic data streams where content changes frequently, you are stuck paying the full tax on every request.”

0.68

Enterprise data lakes are typically measured in terabytes or petabytes, which vastly exceeds what can fit into even the largest LLM context windows (millions of tokens), making it impossible to fit all enterprise data into a single context window.

factualhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Now a context window of millions of tokens sounds great but in the scheme of enterprise data that's really just a drop in the bucket. I mean an enterprise data lake that's probably measured in terabytes or or maybe even petabytes.”

0.68

For an infinite dataset (enterprise knowledge at scale), a retrieval layer is necessary to filter information down to something that fits into the LLM context window.

causalhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“So if you want an infinite data set that stores everything, you really do need to have a retrieval layer to filter information down to something that fits into the LLM context window.”

0.68

The massive increase in LLM context window capacity forces architectural reconsideration: if we can fit all documentation into the context window, we must ask whether we really need the overhead of embedding models and vector data stores, raising the question of whether RAG is becoming an unnecessary complexity layer.

causalhigh valuecontestednovelty 2/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“So, this massive jump in capacity forces us to ask a difficult question about our architecture. Because if we can simply command A, command C, command V, all of our documentation into the models context window, do we really need the overhead of embedding models and vector data stores? Is RAG becoming an unnecessary complexity layer?”

0.66

RAG relies on the assumption that the retrieval logic will actually find the right information in the vector database, and this assumption creates a critical failure point.

causalhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Now this works but it does rely on something. It relies on the hope that your retrieval logic actually found the right information in the vector database.”

0.64

Long context is a brute-force approach where documents are put directly into the context window and the model's attention mechanism does the heavy lifting of finding the answer, rather than using a retrieval system.

definitionhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Now this is really the model native solution because you skip the database here and you skip the embedding model. All you do is you take your documents and you just well you put them straight into the context window and then you let the model's attention mechanism actually do the heavy lifting of finding the answer.”

0.64

Large language models are frozen in time and know everything about the world only up until their training cutoff date, and they know absolutely nothing about what happened 5 minutes ago or about private data, internal wikis, or proprietary codebases.

factualhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“There's a fundamental truth about LLMs, large language models. They are frozen in time. They know everything about our world up until their training cutoff date and absolutely nothing about what happened 5 minutes ago. Nor do they know anything about your private data, your internal wikis, your proprietary codebase.”

0.62

RAG is not dead; while long context wins on simplicity, RAG still has legitimate use cases and remains necessary in certain scenarios.

causalhigh valuecontestednovelty 1/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“So, is RAG dead? Is the vector database destined for the museum of things we needed in 2024? Well, not quite because while long context wins on simplicity, RAG still has a place.”

0.60

RAG is fundamentally designed to retrieve what exists—it finds semantic matches between queries and specific text snippets. When the answer lies in what's not explicitly in the database (e.g., gaps between documents, omissions, or comparisons across documents), RAG cannot retrieve the gap and the model never sees the full picture required to spot missing pieces.

causalhigh valuespeaker onlynovelty 3/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“A RAG is fundamentally designed to retrieve what exists. It relies on finding a semantic match between your query and a specific snippet of text in your database. But what if the answer lies in what's not in the database? So, so let's say you have a set of product requirements stored as a document and you've also got a set of release notes stored as a document and then we ask which security requirements were omitted from the final release. Now using RAG when you query for omitted security requirements the vector search looks for chunks discussing well security and requirements. It retrieves snippets from the requirements doc. It retrieves snippets from the release notes, but it cannot retrieve the gap between them. And because RAG only shows the model a few isolated snapshots, the model never sees the full picture required to spot the missing pieces.”

0.60

With long context there is no retrieval step, so the model gets to see everything, eliminating the silent failure problem.

factualhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“With long context, there is no retrieval step. The model gets to see everything.”

0.60

Context injection—the problem of getting the right data into an LLM at the right time—has two fundamentally different solution approaches: RAG (retrieval augmented generation), which is an engineering approach, and long context, which is a model-native solution.

definitionhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“And there have been two very different ways to handle this. Now, the first is really what we can think of as the engineering approach. It's RAG, retrieval augmented generation... And the second approach is really a bit more of a brute force approach and that one is called long context. Now this is really the model native solution”

0.60

RAG architecture involves breaking documents into chunks, passing them through an embedding model to convert them into vectors, storing those vectors in a dedicated vector database, performing semantic search to retrieve the most relevant chunks based on user queries, and injecting those chunks into the context window alongside the user prompt.

definitionhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“We take some of the documents that we want to give to this LLM. So these are documents that could be PDFs or code files or entire books and we chunk them. We break them up into smaller chunks and we pass them through to an embedding model and the embedding model takes those chunks and it turns them into vectors and those vectors are then stored in a dedicated vector database. Now when a user asks a question, it performs a semantic search to retrieve the most relevant chunks and then inject them into the context window.”

0.60

Early LLMs had context windows of approximately 4K tokens, which was too small to fit entire documents, forcing reliance on RAG systems.

factualhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“For a long time this kind of brute force method wasn't really much of an option because initially context windows were tiny. Early LLMs had context windows that could maybe store what like 4K of tokens. You couldn't fit a novel in there, let alone a corporate knowledge base. You basically had to use RAG.”

0.60

Long context solves the whole book problem by dumping the full documents (e.g., the complete requirements document and complete release notes) into the context window, allowing the model to perform the full comparison needed to identify omissions.

causalhigh valueestablishednovelty 0/4durability 4/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“The model really needs both of these documents in full to perform the comparison, which is exactly what long context does by dumping the whole book, the full requirements doc and the full release notes into the context window.”

0.50

Modern LLMs have context windows of 1 million tokens or more, which is roughly equivalent to 700,000 words, making it possible to fit the entire Lord of the Rings series plus The Hobbit into a single prompt.

factualhigh valueestablishednovelty 0/4durability 2/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“But today's models have much larger context windows. Some of them have, you know, a million tokens plus. And to put that into perspective, a million tokens is roughly 700,000 words. and you could fit the entire Lord of the Rings series into the prompt and still have room for The Hobbit.”

0.48

If we accept that we can fit whatever data we need into the context window, the primary argument for choosing long context over RAG reduces to a single factor: simplicity.

causalhigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Well, if we accept that we can fit whatever data we need into the context window, then the argument for doing so basically boils down to one word, simplicity.”

0.48

A production RAG system requires multiple infrastructure components: a chunking strategy (fixed size, sliding window, or recursive), an embedding model to encode data, a vector database to store vectors, a reranker to sort results, and systems to keep vectors synchronized with source data, making it a complex system with many potential failure points.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“A production RAG system. Well, it is quite heavy. You need a a chunking strategy which is like fixed size maybe or sliding window or recursive. You decide. You're going to need a embedding model to encode the data. You need a a vector database to store it. You're going to need a reranker to sort the results. you need to keep all the vectors in sync with your source data. It's basically a lot of moving parts, a lot of places for things to break.”

0.48

For problems involving a bounded dataset that requires complex global reasoning (such as analyzing a specific legal contract or summarizing a book), long context is the way to go because it simplifies the stack and improves reasoning.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“Well, if your problem involves a bounded data set and requires complex global reasoning like analyzing a specific legal contract or summarizing a book, I think long context is the way to go. It simplifies the stack and it improves the reasoning.”

0.48

For navigating the infinite dataset of enterprise knowledge, the vector database remains the only viable warehouse for data because the dataset far exceeds what can fit in context windows.

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“But if you're navigating the infinite data set of enterprise knowledge, the vector database remains the only viable warehouse for your data.”

0.47

Long context offers what might be called the 'no stack stack': removing the database, embeddings, and retrieval logic, the architecture simplifies to the minimal problem of getting data to the model.

definitionhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — Is RAG Still Needed? Choosing the Best Approach for LLMs [UabBYexBD4k]

“And long context offers what we might call well just simply the uh the no stack stack. You remove the database, you remove the embeddings, you remove the retrieval logic. The architecture simplifies down to getting the data and just well sending it to the model.”