Cache Storage Options
Where cached answers live
Redis-family stores, server memory and managed edge caches are the usual homes for LLM caches. Match durability, latency and price to your traffic: exact caches love Redis; semantic caches often need a vector index plus metadata store.
What you will learn
- Choose storage by hit-rate vs durability needs
- Pair a fast store with a durable one
- Avoid cache-hot-spot bottlenecks
What is this?
LLM caches need a fast key-value store (Redis/Upstash/Valkey), a quick in-process cache for hottest answers, or a durable store (S3/Postgres) for expensive-to-rebuild responses. Semantic caches usually add a vector index on top of the KV store to find near-duplicates.
Why do you need it?
- An S3-backed full-corpus model answer can be slow to fetch; a Redis layer makes hits feel instant.
- Expensive generations (10K-token outputs) justify durable persistence.
- A single Redis hot-key can bottleneck a fleet; sharding and TTL balance help.
Mental model
Storage tiers
A cook's workspace
Salt is by the stove (in-memory), pantry staples in cupboards (Redis), and rarely-used ingredients in the cellar (S3). You reach for what fits the hand.
How it works
- In-memory shard cache: fastest, dies with the process, small.
- Redis/Upstash: fast network store, TTL native, shared by all replicas.
- Durable store: for cold-start protection of expensive responses; miss→S3→miss→model.
- Semantic tiers: vector index for near-dup lookup, KV for the answer object.
Layered cache lookup.
In production
- A high-QPS FAQ puts top-100 in-memory, full set in Upstash Redis.
- An agent-level cache writes expensive answers to S3 and reads Redis first.
- A vector store holds semantic keys while Redis stores the answer blobs.
Best practices
- Cache where the hot path reads: app-proximate for sub-ms hits.
- Keep L1 small and TTL-short; let L2 own the bulk.
- Persist only expensive rebuilds, and set a storage-size guard.
- Partition keys (per tenant/model) so one noisy key doesn't starve others.
Common mistakes
- Putting everything in-process → cache thrashes on every deploy.
- S3-only cache → cold feels like the cache isn't there.
- One giant Redis without sharding → hot-key and memory bloat.
Debugging tips
- Hit in L2 but slow responds → network/deserialization, not storage choice.
- Memory grows past budget → TTL-less keys; enforce TTL and evict.
- Semantic lookups crawl → vector index missing or brute-force scans.
Interview checkpoint
Quick quiz
What pairs naturally with a semantic cache?
Larger always means better for a cache store.