Deploying Model Updates
Shipping weights like you ship code
Model changes are product risks: a new weights version can silently shift tone, correctness or cost. Canary serving splits traffic, compares on offline + online evals, and rolls back fast when quality regresses.
What you will learn
- Design a canary rollout for a model
- Set go/no-go gates on evals and metrics
- Roll back a model without breaking clients
What is this?
Canary deploys run a new model version on a small slice of traffic alongside the current one. You compare outcomes — offline evals plus online metrics like task success, refusal rates and latency — and only widen the slice when the new version holds the bar, with an instant rollback path.
Why do you need it?
- Provider models and your fine-tunes change on someone else's or your own schedule.
- A subtle quality dip is cheaper to catch on 5% traffic than on all of it.
- Rolling back to old weights is fast; rolling back an unhappy userbase is not.
Mental model
Canary
A bridge test
You walk the loaded bridge with the canary first; if it survives, everyone crosses. Same for traffic slices.
How it works
- Offline gauntlet: run a golden eval of the candidate weights; reject if it regresses.
- Shadow mode: copy a slice of requests to the candidate and compare outputs (no user impact).
- Canary: route 5–25% real traffic to the candidate behind the gateway.
- Promote when metrics hold for a window; otherwise flip weights back via config.
Rollout funnel.
In production
- A provider bumps its flagship model; the org eval-gates before updating the model id mapping.
- A fine-tuned model canaries for 3 days, monitoring refusal and task-success before promote.
- A token-cost-sensitive path canaries a smaller model and keeps it if quality holds.
Best practices
- Freeze a golden eval set per product surface; gate every version change on it.
- Compare apples to apples: same prompts, same sampling on the canary window.
- Monitor per-version metrics: latency, cost, refusal, tool-call correctness.
- Make rollback a config flip, never a redeploy.
Common mistakes
- Rolling out a new model id without running the offline eval — the easy-to-miss regression.
- Comparing versions on different prompts (routing drift, not model change).
- No time-boxed window, so a slow degradation silently reaches full traffic.
Debugging tips
- Quality dip that only shows at rollout time: replay the canary prompts on the old version.
- Latency regression in the canary behind uniform code = weights/precision, not infra.
- Small traffic slices hide rare failures — use them only after the eval gauntlet, not instead of.
Interview checkpoint
Quick quiz
What is shadow traffic in a rollout?
A golden eval that passes guarantees the rollout is safe.