Request Replay and Canaries
Test against real traffic before real users
Request replay feeds a captured production traffic stream into a candidate model to test it before live exposure; canaries then serve a slice. Together they give real-world evidence without risking the whole user base.
What you will learn
- Capture production request streams for replay
- Compare candidate vs champion on replayed traffic
- Layer replay before canary in the rollout sequence
What is this?
Request replay feeds recorded production traffic into a candidate model as if live — capturing inputs and comparing outcomes against the champion — while canaries subsequently route a small live slice. Both provide real-traffic evidence with capped risk.
Why do you need it?
- Synthetic load misses the long tail of real requests.
- Replay is zero-risk: the candidate never sees a user.
- Canary validates live integration (latency, features, outcomes) with bounded blast radius.
- A replay+canary sequence replaces 'works in tests → prod' blind jumps.
Mental model
Civil aviation uses the simulator before a real passenger flight
Replay then canary
Pilots train on recorded conditions (replay) with no passengers; then a low-passenger real route (canary) proves the integration before the full schedule. Both gate the big step.
How it works
- A capture layer logs anonymized production requests (features, timestamps) to storage.
- The replay harness feeds the capture to the candidate in a sandbox and records outputs.
- Diff metrics compare candidate vs champion outputs on identical inputs.
- On green diff, a canary routes a live slice to the candidate under monitoring.
- Canary metrics (latency, outcomes, error rate) decide promotion.
Replay-then-canary rollout.
In production
- A fraud candidate is replayed over 24h of captured transactions before any live exposure.
- A rerank model canaries 5% after a clean replay diff, rolling to 100% on metric holds.
- Replay revealed a feature-skew bug between train and serve before users saw it.
Best practices
- Capture a representative, privacy-safe sample with user consent/logic.
- Run replay under realistic concurrency so latency is honest.
- Define the diff contract (metric + threshold) before replaying.
- Keep replay and canary as separate gated steps in CD.
Common mistakes
- Replaying with stale/privileged capture that doesn't match live request mix.
- Comparing only score distributions, ignoring business-sensible outcomes.
- Canary without a replay baseline, so live surprise hides behind 'it was different'.
- PII leakage in captured payloads — scrub before replay touches storage.
Debugging tips
- Replay diff unstable — re-checksum inputs; ordering or nondeterminism pollutes comparisons.
- Canary diverges from replay — live feature availability differs; inspect the join, not the model.
- Captures grow forever — TTL and sampling policies are required infrastructure.
Interview checkpoint
Quick quiz
Replay's user-impact risk is…
Canary validates live integration that replay cannot see.