CD for ML
Shipping a model version, safely, every time
Continuous Deployment for ML moves a validated registered model through stages — staging, canary, production — with automated checks and promotable rollback. It's delivery of an artifact, not a rebuild at the edge.
What you will learn
- Promote a registered model through staging and canary environments
- Use deployment gates and automated smoke validation
- Roll back to a prior registry version through the same path
What is this?
CD for ML promotes a registered model artifact through a sequence of environments — staging, canary, production — with automated validation at each hop and the ability to roll back by pointing at a previous version. The artifact moves; the environment is rebuilt around it or the model is swapped in place.
Why do you need it?
- Direct 'deploy to prod' skips the safety stages that catch integration issues.
- Canaries and staged rollouts bound blast radius when a model misbehaves.
- Rollback via registry versioning is mechanical, not a retrieval of 'yesterday's model'.
- Promotion discipline makes every production model attributable to evidence.
Mental model
A nerve-testing promotion ladder with probation
Model promotion through stages
An employee starts on probation (staging), moves to supervised work (canary) and earns tenure (production) after reviews. One bad performance review ends the climb and they revert to a known-good role (rollback).
How it works
- A registry event marks a model as 'staging' with its evidence bundle.
- Staging deployment boots with the pinned artifact and runs health/probe checks.
- A canary slice routes a percentage of traffic; automated evaluation compares outcomes.
- On passing, the model is marked 'production' and full traffic shifts.
- Each promotion writes a deployment record so rollback targets a known version.
- Incident? CD re-promotes the previous 'production' version mechanically.
Model promotion across CD stages.
In production
- A weekly retrain auto-promotes to staging, canary for a day, then production on green telemetry.
- A serving regression on the canary pauses promotion and re-pins the previous version.
- Aurora-style teams use feature flags with model versioning so both can ship independently.
Best practices
- Promote only registered artifacts, never ad-hoc files.
- Automate staging; gate prod on evidence plus optional approval.
- Make rollback a first-class button pointing at the last production version.
- Separate model rollout from code rollout so you can decouple blame and revert.
Common mistakes
- Redeploying the whole service to 'ship a model' — the artifact should move instead.
- Manual canary: flying 100% traffic because the box was healthy.
- No deployment records, so 'rollback' becomes archaeology.
- Promoting on train metrics alone, no real-traffic evaluation.
Debugging tips
- Pods healthy but canary metrics bad — the model may be fine and the routing/skew is the bug.
- Rollback is slow — pre-warm the previous artifact; don't rebuild on demand.
- Staging passes, prod fails — compare data distributions between the environments.
Interview checkpoint
Hands-on lab
- Simulate a registry with staged promotions
- Enforce evidence thresholds before promotion
- Implement a rollback path to a known-good version
Promote a Model Through Stages
35 min · intermediate
Quick quiz
What is the primary value of a canary stage in model CD?
A production deploy should always record the registry version it came from.