Context window
The maximum amount of text/tokens a model can hold in memory at once when processing a session.
As defined in lex_peter-steinberger
Other definitions
The amount of information (conversation and code) supplied to the model at once; performance degrades when it grows too large.
Vibe Coding Tutorial and Best Practices (Cursor / Windsurf)The maximum length of text (usually measured in tokens) that a language model can consider at once when processing input, including both the retrieved context and the user query.
Knowledge Graph or Vector Database… Which is Better?
Where it comes up · 44
- Is RAG Still Needed? Choosing the Best Approach for LLMs11 claims
- Feed Your OWN Documents to a Local Large Language Model!10 claims
- How Investors Are Using AI7 claims
- A Quick Lesson Of AI Languages7 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job5 claims
- Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI5 claims
- Every Claude Code Workflow Explained (& When to Use Each)4 claims
- Chris Pedregal - Building Granola - [Invest Like the Best, EP.412]3 claims
and 36 more
Most important claims · 10 of 105Search all claims for “Context window” →
- The 'life2vec' project encoded the life stories of about six million people in Denmark (2008-2020) from government statistical records into an artificial symbolic language fed to a transformer model, achieving roughly 78% accuracy at predicting events like whether a person aged 35-65 would die within four years—an age range chosen because it is harder to predict than younger or older ranges.301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
- There is an intuitive assumption that if data is in the context window, the model will use it, but research suggests otherwise: as context windows grow to 500,000 tokens, the model's attention mechanism becomes diluted, and the model often fails to retrieve or hallucinates details from text that is buried in the middle of very large documents.Is RAG Still Needed? Choosing the Best Approach for LLMs
- Information memorized in model parameters is memory-efficient (roughly one fact per parameter) but squishy, leading to hallucination and confused facts because trillions of tokens are churned together; information in the context window is sharp and clear thanks to the attention mechanism, but each token costs far more memory (kilobytes to a megabyte of keys and values per token across layers), creating a fundamental trade-off.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- Uploading documents to a model's context window is like providing a student with a temporary cheat sheet during an exam; the model can reference and use the documents to answer questions within that session, but once the session ends, the information is not retained unless the documents are uploaded again in the next session.Feed Your OWN Documents to a Local Large Language Model!
- Within a context window, every single token counts toward a total token budget for a single interaction, including the user's question, system instructions, background information retrieval, and the generated answer—all competing for the same limited allocation.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
- The KV cache stores a representation of every token in the conversation and often becomes larger than the model weights, and because its size depends on users and context, it is the main limit on purely SRAM-based low-latency serving.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Context management is the third harness primitive; it decides what context matters and enters the model at a given moment. Techniques include retrieval-augmented generation (RAG), reranking of retrieved chunks, summarization at intervals, compaction when context limits are reached, and context compression. The goal is to protect the model's attention by ensuring only relevant context consumes the limited context window.Harness Engineering Masterclass: Technical Deep Dive on how to build Agentic Systems
- Even within projects, AI models have a finite context window, so with very large or numerous files the model may not use every piece of information with perfect recall.STOP Using ChatGPT Normally. Use Projects Like This Instead
- Hallucination can be mitigated by allocating token budget to retrieval of real factual tokens and injecting them into the context window, providing the model with higher-quality 'money' to spend rather than forcing it to guess.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
- When a conversation exceeds the context window limit, the AI begins forgetting earlier parts of the conversation, which is why long chats can go off the rails or become incoherent.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI