Caching
Intermediate 12 min read Caching

Semantic Caching

Cache on meaning, not spelling

Semantic caching keys replies by meaning, not exact text: a near-duplicate question reuses the stored answer when embedding similarity clears a threshold. It saves on repetitive-but-not-identical traffic at the price of a distance computation and collision risk.

semantic cacheembeddingscosine similaritynear-duplicatecollision
Prerequisites: Text Embeddings
Before we start

What you will learn

  • Key a cache by embedding distance
  • Tune a similarity threshold with a collision test
  • Decide when semantic beats exact caching
Introduction

What is this?

A semantic cache embeds each request, then compares it to stored request embeddings. If cosine similarity exceeds a threshold, the stored response is returned as a 'good-enough' match — even when the wording differs. You trade a small chance of mismatch for much higher hit rates.

Motivation

Why do you need it?

  • Users ask the same thing many ways: 'refund policy', 'can I get a refund', 'how do refunds work'.
  • Exact-match keys miss all of those; semantic keys catch the family.
  • One embedding lookup (~ms) beats a 1-2s model call whenever it converts a miss to a hit.
Think about it

Mental model

Imagine

Semantic cache

Think of

A librarian's fuzzy recall

'The tall building book' still gets you 'Skyscrapers: A Visual History' — the librarian matches meaning, not exact titles.

Mechanics

How it works

  • Embed the request; compare against a stored list of prior request embeddings.
  • If max cosine ≥ threshold, return that entry's answer.
  • On miss, embed-save answer, respecting TTL and capacity.
  • Threshold picking is a collision test: too low returns wrong answers.

Similarity gate.

Embed
Nearest
Gate
Miss
Hands-on

Examples

A similarity check

A similarity check
import numpy as np
 
def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
q = embed("how do refunds work")
stored = [{"q": embed("refund policy"), "a": "..."}]
best = max((cosine(q, e["q"]), e) for e in stored)
hit = best[0] >= 0.92
print("hit" if hit else "miss", round(best[0], 3))

Crossing the gate serves the stored answer; beneath it the model runs again.

Real world

In production

  • A customer FAQ answers near-duplicates from one cache entry, hitting 70%+.
  • A code helper reuses similar-snippet answers, cutting burn on repeated themes.
  • A lead-generation bot caches common intent clusters instead of every phrasing.
Guidelines

Best practices

  • Set the threshold high initially (0.9+) and lower it with measured collisions.
  • Store (request_embedding, answer) pairs, not just answers.
  • Combine with exact-key first: exact hits are free and perfectly safe.
  • Sample collisions periodically — sample real traffic to tune.
Watch out

Common mistakes

  • Threshold too low → wrong answers presented confidently.
  • Storing stale answers under near-duplicates forever (no TTL).
  • Forgetting the model/params in the key semantics.
When things break

Debugging tips

  • Wrong matches when threshold passes → lower... no, RAISE the threshold or split the corpus by intent.
  • Hit rate flat vs exact cache → your traffic has no near-duplicates; revert to exact.
  • Slowness from scanning many vectors → index the embedding store or cap candidates.
Interviews

Interview checkpoint

Practice

Hands-on lab

Gate a Cache by Similarity
35 min · intermediate
In progress
Objectives
  • Represent requests with normalized keyword vectors
  • Compute cosine similarity between requests
  • Serve near-duplicates above a threshold

Gate a Cache by Similarity

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Semantic caching decides a hit using what?

2

Lowering the similarity threshold always raises hit rate safely.

Go deeper

Official documentation

Cosine similarity
docs
Annoy / HNSW indexes
docs