A/B and Shadow Evaluation
Proving the challenger on real traffic
Shadow serving runs a candidate on real traffic without affecting outcomes — the gold standard for low-risk evaluation; A/B/A-B testing turns live slices into statistical inference with guardrails and stopping rules.
What you will learn
- Run shadow deployments to evaluate without user impact
- Design A/B sessions with significance and guardrail discipline
- Decide rollout via evidence instead of faith
What is this?
Shadow and A/B evaluation put challengers in production traffic but hold the decision: shadow serving logs candidate predictions into a replay/evaluation store without affecting users; A/B testing splits live population into statistically sized, guardrailed experiments whose outcomes release or reject the challenger.
Why do you need it?
- Shadow is the safest loudspeaker — real inputs, zero user exposure.
- A/B converts anecdotes into significance-gated decisions.
- Both catch what offline and proxy evaluation miss.
- Guardrails stop a lift in one metric being a rot in another.
Mental model
A stand-in driver in a simulator vs a real road test
Shadow vs A/B evaluation
First the challenger 'drives' a replica of the road (shadow) — same traffic, no passengers — to see if it crashes. Then, if it holds, a small stretch of the real road where you measure with stoplights and a control group (A/B). Both cheap ways to avoid a real crash.
How it works
- Shadow: route live traffic to champion AND challenger; challenger's decisions are logged, not served.
- Replay those shadow decisions against stored outcomes to score the candidate.
- A/B: random assignment (or bandit allocation), power-calc sizes, pre-registered metrics.
- Guardrails: cost, latency, business-safety metrics move or block the experiment.
- Ships only when significance and guardrails both pass; rollback armed.
Shadow to A/B rollout.
In production
- A Fraud team shadows a challenger for two weeks, replays approved/declined against settled fraud, then A/B's the winner.
- A Recommender A/B's ranking changes on 5% traffic with GMV + engagement guardrails.
- A Search team shadows new embeddings to build an offline-quality release case before any live experiment.
Best practices
- Shadow on the production input stream with stored IDs for replay scoring.
- Pre-register A/B metric, size, duration, and stopping rules before the flip.
- Guardrail on cost/latency/safety metrics alongside the primary.
- Automate the ramp; kill-switch-first is second nature.
Common mistakes
- A/B'ing without a power analysis — underpowered 'proof'.
- Peeking and stopping on early significance — inflating false positives.
- Shadow decisions indistinguishable in logs — replay scores the wrong bucket.
- A single primary metric flying, blind to guardrail damage elsewhere.
Debugging tips
- Shadow score mismatch with live — check the routing flag actually shadowed it and IDs align.
- Small A/B 'significance' that flips weekly — fix sample size; you're peeking at variance.
- Guardrail trips on rollout — isolate the metric that moved; often it's cost/latency, not effect.
Interview checkpoint
Hands-on lab
- Route live traffic to a shadow challenger
- Replay shadow decisions against stored outcomes
- Compare champion vs challenger with a significance guard
Shadow-Replay the Challenger
45 min · advanced
Quick quiz
Shadow serving differs from A/B because its decisions…
Regularly peeking at your A/B result and stopping when it looks significant is good practice.