Caching for Cost
One paid answer, served many times
Exact-key caching and provider prompt-cache discounts cut repeated spend; semantic caching collapses near-duplicate requests. Each has a correctness cost — staleness — so pair caching with careful TTLs and versioned keys.
What you will learn
- Choose exact vs semantic caching per workload
- Leverage provider prompt-cache discounts
- Design invalidation to bound staleness
What is this?
Cost caching has three layers: exact-key caching for identical requests, semantic caching for near-duplicates, and provider prompt caching that discounts repeated prefixes inside greedy contexts. The cost win compounds with heavy reuse, but stale content is the tax you pay — mitigate with TTLs and invalidation.
Why do you need it?
- Identical or near-identical requests (support FAQs, daily digests) pay full price each time otherwise.
- Simple exact caches catch exact repeats but miss paraphrases.
- Long shared system prefix means provider cache-read pricing nearly eliminates that part of the prompt bill.
Mental model
Caching for cost
A shared water cooler
Everyone drinks the same cold water instead of firing up the boiler per sip — but you must purge the tank when the recipe changes.
How it works
- Exact cache: hash inputs + model + params; serve identical calls for free.
- Semantic cache: vector similarity gates near-identical prompts to a stored answer.
- Provider prompt cache: keep the stable system/context prefix; long-lived prefix reads bill at a steep discount.
- Invalidate by version bump or TTL so nobody sips stale answers too long.
Cache decision flow.
In production
- An FAQ bot returns cached answers for 80% of hits with a 10-minute TTL.
- A product copilot uses provider prompt caching so the giant system prompt is billed at ~10%.
- A news digest semantic-caches identical-sounding daily queries.
Best practices
- Keep cache keys versioned so model/prompt changes invalidate cleanly.
- Set the TTL to the staleness your product tolerates.
- Count cache-hit rate as a metric; it's the health of the cost lever.
- Compose: exact cache handles repeats, prompt cache discounts prefix, semantic catches paraphrases.
Common mistakes
- Semantic-caching a request that's time-sensitive (prices, live scores) → stale answers.
- Cache keys that ignore model or params → wrong-tier answers served from cache.
- No invalidation after a prompt redeploy → old logic sticks around.
Debugging tips
- Hit rate near 100% and content stale → TTL or invalidation too lax; check update frequency.
- Semantic hits on 'new' queries → threshold too low; same phrase noun collides.
- Provider discounts not showing → prefix varies per request; canonicalize the shared preamble.
Interview checkpoint
Hands-on lab
- Trim the cheapest redundancy first
- Apply exact and similarity caching
- Model provider cache-read pricing
Cuts, Caches and Discounted Reads
30 min · intermediate
Quick quiz
Semantic caching is best when?
Provider prompt caching bills recurring prefixes at a discount.