Online Testing: Shadow, Canary & A/B

• Compare shadow, canary and A/B evaluation on real traffic
• Design traffic-split experiments with power and runtime
• Choose the technique from the risk and the question you're asking

• Curated offline data can't represent live distributions; only real traffic can.
• Business impact (revenue, retention) is rarely the metric you optimized offline — A/B measures the thing your model exists for.
• Each technique trades risk for signal: shadow is safe but indirect; canary risks a slice; A/B measures causal impact.
• Incremental rigor beats 'it looked better in the notebook' for every promotion decision.

A wind tunnel (offline eval) predicts a lot but can't test real turbulence and pilots. A first flight with a chase plane (shadow) gathers real data safely. A small fleet rollout (canary) finds what the chase plane missed. Full certification with routes and passengers (A/B at scale) validates the economics. No team skips the ladder and still calls the plane ready.

• Shadow: duplicate traffic to a candidate, log its scores, serve the incumbent's decision; diff candidate vs incumbent distributions.
• Canary: route X% of live requests to the candidate; compare latency/errors/proxies; promote or roll back on thresholds.
• A/B: stable random assignment of users/models; run long enough for power; measure the primary business metric plus guardrail metrics.
• Loop variant: A/B is also good for decision-threshold experiments without touching the model weights.
• Gate every promotion on a pre-registered decision rule so analysts choose before results load.

• A recommender shadows candidate v2 for a week, abandons it when score distributions diverge too far, and only then canaries 5%.
• A search team A/B tests a reranker on 10% of users, gated on CTR and p95 latency as the guardrail.
• A pricing model canaries by region, rolling back instantly when revenue-per-request dips on the flagged cohort.

• Go shadow → canary → A/B in order of evidence need and risk appetite; never skip straight to 100%.
• Pre-register the decision rule: effect size, metric, power and run length before the experiment starts.
• Keep guardrail metrics (latency, error rate, other business KPIs) monitored during every rollout phase.
• Automate canary promotion/rollback thresholds so humans review, not babysit.
• Log consistency between shadow and live so you can audit 'would the candidate have changed this decision'.

• Running one A/B, piecing it, and promoting without power — then attributing the next week's noise to the model.
• Using A/B for a recommendation change whose unit isn't independently assignable (network effects break the assumption).
• Canary rolling so fast there's no window to detect latency regressions before 100%.
• Shadowing without storing decisions — missing the whole evidential value.

• Diverge-score distributions between shadow and live; a candidate that never changes decisions is dead weight or a bug.
• Watch guardrail latency during the first canary hours; cold-start caching differences show up there.
• For A/B, verify assignment balance (no Simpson-style confounds) before reading results.
• If an A/B shows a win but the shadow diff looked calm, check whether the change touches decision thresholds, not weights.