Cache Invalidation
Keeping cached answers honest
Every cache is a time bomb of stale answers unless invalidation is explicit. TTL, version keys and event-driven purges keep cached LLM responses honest when products, prices or facts change.
What you will learn
- List invalidation strategies and their tradeoffs
- Version cache keys on model/config changes
- Design events that purge derived answers
What is this?
Invalidation is how a cache forgets: TTL (time-based), version keys (build-based) or purge events (change-based). For LLM apps, staleness matters because an answer that referenced the old pricing or old model becomes a support disaster after a deploy.
Why do you need it?
- Answers go stale when the world changes (prices, policies, product facts).
- TTLs alone can't coordinate with deploys — version keys can.
- Purge events matter for facts that invalidate many keys at once (a doc re-baseline).
Mental model
Invalidation
A milk carton's date
The carton (TTL) protects you from unseen age; a heat wave (event) can spoil it early and needs its own signal.
How it works
- TTL: each key expires after N seconds; simplest, bounded staleness.
- Version key: bake a version into the key (model_id + content_version); bumping the version orphans old entries.
- Purge events: on content/model deploy, delete keys in affected namespaces.
- Soft invalidation (TTL) plus hard invalidation (deploy hook) overlap for safety.
Three invalidation signals.
In production
- A pricing bot bumps the key version on every price deploy — no stale quotes.
- A doc assistant purges its namespace when documents re-index.
- A model upgrade flips responses: gateway changes model_version in the key automatically.
Best practices
- Always TTL, even when you think content never changes.
- Include product/data version in the cache key at the namespace level.
- Wire deploys to flush affected namespaces (purge to zero, then rebuild).
- Monitor staleness ticket rate as the invalidation health metric.
Common mistakes
- Permanent cache on facts that flip overnight.
- Version bumps that don't actually change the keys' namespaces.
- Purging everything on every deploy — thundering herd plus cold start.
Debugging tips
- Stale answers persist after change → version key not bumped, or TTL longer than deploy cycle.
- Rebuild storms post-purge → stagger rebuilds or keep a warm small TTL.
- Some users see old answers → per-region or per-replica cache divergence; centralize.
Interview checkpoint
Hands-on lab
- Bake a deployment version into cache keys
- Purge a namespace atomically
- Observe stale answers disappear
Version-Key Invalidation
25 min · intermediate
Quick quiz
Which invalidation matches a code deploy best?
TTL alone makes a cache safe from deploy-driven staleness.