token budget
The total allocation of tokens available within a context window for a single interaction, which must be divided among the user's query, system instructions, background information retrieval, and the generated response.
As defined in Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
Where it comes up · 1
- Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI3 claims
Most important claims · 3 of 3Search all claims for “token budget” →
- Within a context window, every single token counts toward a total token budget for a single interaction, including the user's question, system instructions, background information retrieval, and the generated answer—all competing for the same limited allocation.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
- Hallucination can be mitigated by allocating token budget to retrieval of real factual tokens and injecting them into the context window, providing the model with higher-quality 'money' to spend rather than forcing it to guess.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
- The retriever component searches a database for tokens mathematically similar to those in the user's query (by converting them to vectors) in order to inject relevant factual information into the token budget.Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI