Caching
Beginner 10 min read Caching

Response Caching

Same question, already-answered

Response caching stores exact LLM outputs for identical requests so repeat calls return instantly for free. It's the cheapest latency and cost win in AI apps — if your prompts are cacheable and your answers tolerate staleness.

cacheidempotentTTLexact matchcost saving
Prerequisites: Token Economics
Before we start

What you will learn

  • Recognize which LLM calls are cacheable
  • Choose a cache key and a TTL
  • Add a cache without changing correctness
Introduction

What is this?

Response caching saves the response to an exact prompt (system + user + config) and returns it on the next identical request, skipping the model call entirely. It works when the same question recurs and the answer staying fixed for a while is acceptable.

Motivation

Why do you need it?

  • Identical traffic is pure waste: FAQs, top-N queries and repeated batch rows hit the model each time.
  • Each hit removes latency (no model call), cost (no tokens) and load (no GPU/API work).
  • Perfect keys let caches be dead-simple: hash the full request, compare byte-for-byte.
Think about it

Mental model

Imagine

Response cache

Think of

A barista's regular's card

The regular's usual order is pre-made and handed over instantly; a first-timer still waits for the full process.

Mechanics

How it works

  • Compute a key from the exact request (system prompt + messages + temperature + model).
  • Look up the key in a store (Redis/Upstash); on hit, return the stored response.
  • On miss, call the model, store the response with a TTL, and return it.
  • Streaming responses cache the full body when complete; middleware can handle both.

Hit vs miss path.

Request
Lookup
Miss
Serve
Hands-on

Examples

A cache key and TTL

A cache key and TTL
cache_key = hash_fn(
    model, temperature,
    [m["content"] for m in messages]
)
cached = redis.get(cache_key)
if cached:
    return cached
result = call_model(...)
redis.set(cache_key, result, ex=3600)  # TTL 1h
return result

An exact hit skips the model; TTL bounds staleness.

Real world

In production

  • A support bot caches answers to the top 50 questions for an hour.
  • A docs explainer caches exact repeat queries, slashing spend by half.
  • A batch job that re-runs daily caches unchanged rows between runs.
Guidelines

Best practices

  • Include model, temperature and sampling in the key — not just the text.
  • Set TTL, not permanent: answers age, pricing and features change.
  • Cache idempotent, anonymous or coarse-grained requests first.
  • Instrument hit rate and expose it with the tracing ID.
Watch out

Common mistakes

  • Caching personalized, non-idempotent requests and serving stale private data.
  • A key that ignores model/params → wrong answers for different configs.
  • No TTL: a cached answer forever contradicts the product after updates.
When things break

Debugging tips

  • Hit rate near 0 → key includes volatile noise (timestamps, ids); normalize it.
  • Wrong cached answer for same-looking question → key misses a semantic bit (user tier).
  • TTL too long → stale-content support tickets; shorten and re-eval.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build an Exact-Key Cache
25 min · beginner
In progress
Objectives
  • Implement a hash-keyed cache for a mock model
  • Serve identical requests from the cache
  • Apply a TTL and observe expiration

Build an Exact-Key Cache

25 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The cache key must include which of these?

2

Recommendation caches should bypass the key entirely.

Go deeper

Official documentation

Redis commands
docs
Upstash Redis
docs