Context Windows & Limits
Everything you can say in one meeting
The context window bounds the tokens a model can attend to at once. Attention cost grows with length, quality degrades mid-window, and budgeting context is a core engineering skill.
What you will learn
- Define a context window and its costs
- List degradation behaviors in long contexts
- Budget context like tokens were money
What is this?
A context window is the maximum token count (prompt + generated so far) a model attends to in one call. More context = more to answer from, but attention cost grows superlinearly with length, and quality in the middle of very long contexts measurably degrades — the famous 'lost in the middle' effect. Smart products manage context as a scarce budget.
Why do you need it?
- Context size multiplies per-call token usage and latency.
- Long-prompt degradation is real; don't trust it blindly.
- Where you place critical instructions changes whether the model follows them.
- Context engineering (retrieval, summarization, trimming) is the heart of RAG and agents.
Mental model
A whiteboard that runs out of space
The context window
You can write a lot on the whiteboard, but the near-erased corners get ignored. Important reminders travel best when they're big, recent, and near the front — attention forgets the middle of a truly jammed board.
How it works
- The system writes the prompt; every generated token appends to the same context.
- Attention cost grows ≈O(N²) in sequence length, so long contexts are expensive.
- Positional encoding places tokens in order; reordering changes meaning to the model.
- Very long contexts often show degraded recall for middle content.
- Tokens escaping the window get truncated — data vanishes unless you manage it.
Where quality lives in a long prompt.
In production
- A RAG callstuff the retrieved chunks near the start/end of context, not the middle.
- Agents summarize old turns into one 'history' block to respect the window.
- Docs assistants map classic 'chunk → query → pick top-k' within the budget.
Best practices
- Set real token budgets: fixed cap for system+tools, flexible for retrieved content.
- Place the most recent or most important instructions at the edges.
- Measure actual recall on your long-context tasks instead of assuming.
- Trim history with summaries; never let the window silently fill.
Common mistakes
- Assuming 'huge window' means degradation vanishes.
- Filling context with stale retrieval and paying heavier latency.
- Truncating from the tail — older-but-important instructions disappear.
- Letting agent turns grow unboundedly until the next call 'fits (barely)'.
Common issues
Model forgets instructions from the middle of a long promptlost-in-the-middle attention degradation
Move critical instructions to the edges; shrink context; verify with explicit tests.
Truncated context loses the point of the callToken overflow silently cutting the prompt tail
Budget before the call: count tokens, trim summaries, drop lowest-value content.
Debugging tips
- A model ignores a rule that existed mid-prompt → move it to the start or end.
- Answers degrade as tools pile up logs → summarize or drop low-value tool output.
- Slow calls with long prompts → context is the dominant latency; trim it.
Interview checkpoint
Quick quiz
Where do models usually recall content most reliably?
Longer context is always monotonically better for quality.