Inference Optimization
Intermediate 11 min read Inference Optimization

Prompt and Prefix Caching

Reuse the prefix, skip the prefill

When the same prompt prefix repeats across requests (system prompts, tool definitions, instructions), caching that prefix's KV state skips most prefill work and cuts TTFT and cost. It's a mechanical, high-value win for chat, agents and multi-turn apps.

prefix cachingprompt cachingKV cacheTTFTcache hit
Prerequisites: Throughput Tuning
Before we start

What you will learn

  • Explain what a prefix cache stores
  • Identify prompts that benefit
  • Structure prompts so prefixes actually repeat
Introduction

What is this?

Prefix caching keeps the attention key/value state for tokens that reappear at the start of many requests (the system prompt, tool schemas, a long instruction block). A cache hit skips the prefill for that prefix, so TTFT drops dramatically and the same tokens get billed less.

Motivation

Why do you need it?

  • Prefill is compute-heavy; a 10K-token system prefix recosted every turn is pure waste.
  • Multi-turn chats repeat the whole history on every turn — cache primes help a lot.
  • Provider-side caching lowers both latency and price, with zero client code when enabled.
Think about it

Mental model

Imagine

Prefix caching

Think of

A chef prepping mise-en-place

Dice the shared ingredients once in the morning; every dish that turn needs them skips the chopping.

Mechanics

How it works

  • The server keys KV state by matching exact token prefixes between requests.
  • On cache hit, generation starts from the cached prefix instead of recomputing it.
  • Cache entries expire (idle TTL) and share a memory pool — size it mindfully.
  • Providers expose cache hits in usage fields (cache_creation/cache_read) for billing verification.

Hits skip prefill.

Prompt A
Hash prefix
Hit
Generate
Hands-on

Examples

A cache-friendy prompt layout

A cache-friendy prompt layout
messages = [
  { "role": "system", "content": LONG_SYSTEM_BLOCK },  # stable prefix
  { "role": "user", "content": new_question },          # changes each turn
]
# don't inline the system block into every user turn — keep it a shared prefix

Stable content stays a prefix; volatile content sits after it so the reusable state is contiguous.

Real world

In production

  • An agent with a 8K system prompt sees TTFT drop from seconds to ms after the first call.
  • A RAG app caches the shared instruction + tool schemas, recomputing only the query suffix.
  • A batch job repeats identical headers across rows, letting the cache amortize across the run.
Guidelines

Best practices

  • Order prompts so stable blocks come first (prefix, not suffix).
  • Cache size the pool for your concurrent unique prefixes; reaping churn erases the win.
  • Instrument cache hits/reads per request and include them in tracing.
  • Re-check cache eligibility when you change a tool schema (the prefix changes).
Watch out

Common mistakes

  • Interleaving volatile content before stable blocks, forcing a prefix miss.
  • Assuming hits forever: provider ttls recycle idle entries.
  • Counting cache as free memory — it competes with batch/KV budgets.
When things break

Debugging tips

  • TTFT still slow with seeming hits → prefix differs by a token; compare token ids.
  • Usage shows cache_read 0 → prompt ordering or provider TTL is missing the match.
  • Odd token-boundary changes (new system block) burst cache misses — expect a spike after edits.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A prefix cache hit mainly reduces what?

2

Reordering your system prompt can break cache hits.

Go deeper

Official documentation

Anthropic prompt caching
docs
OpenAI prompt caching
docs