Tokenization
Intermediate 12 min read Tokenization

Token Budgets

Spending your context window deliberately

A token budget splits the context window among system, tools, history, retrieval and output — before you send the call. Budgeting is a design skill, not an accident of prompt length.

token budgetcontext budgettoken limitsslotscontext management
Prerequisites: Token Costs & Counting
Before we start

What you will learn

  • Allocate context slots explicitly
  • Protect the output budget
  • Trim history and retrieval by value
Introduction

What is this?

A token budget reserves parts of the context window for each job: fixed system+tool slot, a history slot that shrinks by compression, a retrieval slot refilled per query, and a guaranteed output slot. Enforcing the budget before sending prevents silent truncation, cost blowouts and degraded answers.

Motivation

Why do you need it?

  • Output truncation silently degrades products and confuses tests.
  • Unbounded history grows every call's cost and latency.
  • Budgeting forces you to choose what matters — the essence of context engineering.
  • Predictable budgets make costs and latencies forecastable, not surprising.
Think about it

Mental model

Imagine

Packing a suitcase with a weight limit

Think of

Token budgets

You can't take everything; the scale (token count) decides. Essentials (system rules, output space) go in first, souvenirs (old chat, redundant retrieval) get compressed or left home. Pack once, fly everywhere.

Mechanics

How it works

  • Define slots: system+tools (fixed), conversation (compressed), retrieval (reset), output (reserved).
  • Count actual tokens per slot with the real tokenizer.
  • Compress old turns into summaries that age out of the budget.
  • Trim lowest-value content before trimming the output budget.
  • Monitor actual vs budgeted usage each call.

A sensible context budget.

Fixed
History
Retrieval
Output
Real world

In production

  • Agents recompute budgets per turn so tool outputs never eat the answer slot.
  • RAG services size top-k retrieval by the remaining retrieval slot.
  • Support copilots compress multi-turn histories into slot-bounded summaries.
Guidelines

Best practices

  • Always reserve an output floor (e.g. 25% or an explicit cap).
  • Enforce budget in code — assert before calling, not after.
  • Log used-vs-budget per slot for calibration.
  • Re-balance slots when adding a new tool or retrieval source.
Watch out

Common mistakes

  • Letting accumulated conversation history silently dominate the window.
  • Shrinking the output slot to nothing for 'more context' — truncation returns.
  • Budgeting by character count.
  • Per-branch budgets that differ across runs (non-deterministic tests).
Root cause analysis

Common issues

Error
Truncated completions at the same ceiling
Root cause

Output slot squeezed out by context growth

Fix

Set an explicit output reserve and enforce it in the assembler.

Error
Summarized history reduces quality
Root cause

Over-aggressive compression

Fix

Keep recent turns verbatim; summarize only old ones; retain named entities.

When things break

Debugging tips

  • Answers cutting off at same token count → output budget starvation; check the reserve.
  • First-answer quality dips after long chats → history compression is dropping kindergarten nuance.
  • Cost creeping per turn → history slot is growing fat; compress earlier.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which slot should you protect from starvation above all?

2

Budgeting by characters is a sound practice.

Go deeper

Official documentation

Managing context (Anthropic engineering blog)
docs