Rollback and Multi-Version Serving
Make 'undo' a mechanical, safe operation
Rollback is a pointer change when artifacts are immutable and versions are registered; multi-version serving (canary, shadow, A/B) is the infrastructure that makes safe release possible.
What you will learn
- Explain why immutable versions make rollback trivial
- Route traffic across multiple live versions
- Design rollback triggers from monitoring signals
What is this?
Multi-version serving runs several model versions simultaneously with weighted traffic routing — champion, canary, shadowing challengers. Because versions are immutable and addressable, rollback is a routing decision: switch the pointer from the new version back to the previous one, instantly and safely.
Why do you need it?
- You can't 'release and forget' a model that might decay; you need an exit stair.
- A canary limits blast radius to a few percent of traffic while evidence accumulates.
- Shadow serving lets a candidate score real traffic without any decision impact.
- Rollback becomes a routing toggle, not a redeploy of old bytes.
Mental model
Airlines swapping runways
Traffic routing between model versions
Planes route to whichever runway is safe NOW. Old runway (old version) remains serviceable and the controller can redirect instantly when weather changes. No runway is ever 'deleted' — it's just not the active choice.
How it works
- The serving router holds a routing table: version → weight, currently serving champion at 100%.
- A release shifts weight gradually (5% canary → 25% → 100%) as evidence gates pass.
- Shadow mode scores the challenger on copied requests with no impact; results compare to the champion.
- Monitoring the canary compares metrics (latency, error, drift, business proxies) to thresholds.
- A threshold breach triggers an automatic rollback: routing flips back to the champion version.
Graduated rollout with automatic rollback.
In production
- A fraud team promotes a nightly candidate to 10% canary; a 2× error-rate spike auto-rolls back within minutes.
- A recommendation challenger shadows live traffic for a week, and only promotes after revenue proxy parity.
- A lending model ships as new MAJOR, keeping the old version ready so a revalidation failure reverts instantly.
Best practices
- Keep at least the prior two versions registered and routable.
- Instrument every request with the serving version so monitoring can attribute.
- Define rollback triggers before release: thresholds for error rate, latency and drift.
- Automate canary promotion AND rollback; humans review the exceptions.
Common mistakes
- Canary weight set but no watching metric, so the canary silently misbehaves at 100%.
- Deleting the previous version 'to save space', voiding rollback ability.
- Rollback only invoked manually on a weekend at 3am.
- Version labels missing from requests, so monitoring can't attribute errors.
Debugging tips
- 'Canary broken' — check routing weights and which version served the failing requests.
- Rollback didn't help? Confirm the old version is actually the one now receiving 100%.
Interview checkpoint
Hands-on lab
- Implement a weighted router across two model versions
- Simulate a canary gate watching a metric
- Trigger an automatic rollback from monitoring
Route and Roll Back Between Versions
30 min · advanced
Quick quiz
What satisfies a safe canary rollout?
Your canary's error rate triples 10 minutes after 10% rollout. What's the correct response?