Caching
Advanced 12 min read Caching

Choosing a Caching Strategy

Matching cache type to traffic shape

Exact, semantic, prefix and sub-call caches serve different traffic shapes. Strategy is a portfolio: cache the repeatable core exactly, near-duplicates semantically, and long prefix chrome structurally. Choose by how your requests cluster.

cache strategyexactsemanticprefix cacheportfolio
Prerequisites: Response Caching Semantic Caching
Before we start

What you will learn

  • Classify traffic into cacheable families
  • Map family to cache technique
  • Build a layered cache portfolio
Introduction

What is this?

A caching strategy is a policy for which requests to serve from where: exact-match keys for byte-identical repeats, semantic gates for paraphrases, prefix caching for shared-foreground prompts, and sub-call caches inside pipelines (RAG fetches, tool results). Real apps layer several simultaneously.

Motivation

Why do you need it?

  • Single-technique caches either miss too much (exact-only) or risk correctness (semantic-only).
  • The traffic mix decides: FAQ shapes love semantic; batch jobs love exact/full; agents love prefix.
  • A layered portfolio converts the most expensive requests with the least risk.
Think about it

Mental model

Imagine

Strategy

Think of

A hospital's triage

Not every patient needs the ICU: triage routes to ER, ward or at-home care — matching response to need and cost.

Mechanics

How it works

  • Sample requests, cluster them by shape (identical, near-dup, shared prefix, unique).
  • Exact hit first (free, safe), semantic second (families), prefix third (structural reuse).
  • Decide eviction/TTL per family; unique traffic is written off.
  • Re-measure hit rate and staleness quarterly as the mix drifts.

Multi-tier gate order.

Exact
Semantic
Prefix
Model
Real world

In production

  • A customer-support app: exact top-100 + semantic family cache + prompt prefix caching.
  • A code assistant caches exact and near-dup queries but never prefixes with live code context.
  • A document QA caches RAG sub-calls (chunk fetches) plus final answer exact keys.
Guidelines

Best practices

  • Measure the traffic mix before designing; cache for the share that repeats.
  • Layer gates cheapest-first and most-correct-first.
  • Set explicit TTL/per-family and review per launch.
  • Keep the model path instrumented so strategy changes are provable.
Watch out

Common mistakes

  • Building the fanciest cache for a traffic shape that's mostly unique.
  • Semantic cache before exact; you pay embeddings for byte-identical hits.
  • No re-evaluation: months later the mix changed but the strategy didn't.
When things break

Debugging tips

  • Semantic catches near-dups but exact rate should still lead; if not, exact key is too noisy.
  • Prefix cache hits with high TTFT → engine-level, check KV pool size.
  • Model spend flat despite high hit rate → hits are the cheap calls, misses the expensive ones.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The cheapest cache layer to check first is usually?

2

One carefully chosen cache technique fits all traffic.

Go deeper

Official documentation

Anthropic caching docs
docs
Cloudflare caching strategy
docs