Prompt Compression
Smaller prompt, same answer
Every prompt token is paid on every request. Compression cuts redundant context — truncating history, summarizing transcripts, selecting and rewriting retrieval snippets — to shrink spend while a preview LLM rewrites dense context into a smaller, still-useful form.
What you will learn
- Identify the redundant parts of a prompt
- Apply deterministic compression and truncation
- Use model-based compression with quality checks
What is this?
Prompt compression shrinks what you send to the model — dropping neutral history, summarizing chat, selecting only relevant retrieval chunks, even using a small model (or LLMLingua-style tooling) to compress verbose text before the main call. The saved tokens are pure profit, but you pay a small preview cost and can damage recall.
Why do you need it?
- Prompt tokens are paid on every single request, so cuts compound.
- Long chat transcripts accumulate irrelevant turns; summaries preserve the gist.
- Retrieval returns whole paragraphs when one sentence would do.
Mental model
Compression
Editing before printing
A copywriter trims a long brief to its essentials for the designer — same intent, a fraction of the ink.
How it works
- Deterministic: truncate oldest turns, cap history, keep the recent decision points.
- Summarize multi-turn or long-doc context with a cheap model (or a summary tool).
- Select retrieval chunks by relevance; optionally rewrite/condense them.
- Verify with evals that compression doesn't drop facts the answer depends on.
Prompt pipeline.
In production
- A sales assistant keeps a rolling summary plus the last three turns instead of full history.
- A support bot sends two relevant chunks instead of ten.
- An email summarizer condenses long threads to one short digest before final answers.
Best practices
- Cut deterministically first (cheap), then decide if model-based compression is worth it.
- Budget the preview-compression cost against the tokens it saves.
- Keep the recent message verbatim; compress age, not recency.
- Run recall evals so compression never silently strips the deciding fact.
Common mistakes
- Compressing the latest user intent along with the old stuff.
- Losing identifiers, numbers and names that retrieval consumers autocomplete from.
- Paying frontier price for the preview when a small model compresses fine.
Debugging tips
- Answers lose specificity → the compressor dropped IDs or numbers; test those on your set.
- Savings vanish → compression preview cost exceeds saved tokens; do it statically.
- Fresh facts missing → truncation removed recent turns; protect their window.
Interview checkpoint
Quick quiz
The safest thing to cut first?
Compression previews cost tokens too.