Model Deployment
Intermediate 14 min read Model Deployment

Model Registry & Promotion

Governed promotion, not copy-paste deploys

A registry is the governance layer between 'good model' and 'live model': versioned artifacts, promotion gates (staging → production), shadowing, and rollback — so every deploy is a decision with lineage, not an act of hope.

registrypromotionstagingshadow deployrollbackgatelineage
Prerequisites: Model Serialization Experiment Tracking
Before we start

What you will learn

  • Model registry stages and their promotion paths
  • Gate promotions with tests and shadow evidence
  • Design rollback as a registry-level operation
Introduction

What is this?

A model registry manages the lifecycle of MODEL versions: each trained artifact is registered with metadata (metrics, commit, data hash), tested, promoted through stages (staging → production) behind gates, and auditable at every level. Shadowing runs a candidate alongside production to collect evidence before cutover. Rollback = repoint production to the previous version — no rebuild, no redeploy of code.

Motivation

Why do you need it?

  • Promotion becomes repeatable evidence, not tribal lore.
  • Shadowing gives before-switch proof that the candidate behaves in traffic.
  • Registry lineage answers 'what is live, who approved it, on what evidence'.
  • Rollback from any regression is minutes away, turning risky changes reversible.
Think about it

Mental model

Imagine

Airports: gates, security checks and a manifest

Think of

Model registry promotion

A flight doesn't just 'start' at the destination — it clears check-in, security, boarding (tests/gates), and only then pushes for takeoff. If the weather turns at the destination, the manifest repoints to the last-safe flight (rollback). Every leg is logged; someone can always say what flew and why.

Mechanics

How it works

  • Register: push trained artifact with metadata; immutable version assigned.
  • Test gate: run evaluation suite, schema tests, and compare vs current production metric.
  • Stage: promote to staging; deploy to a staging endpoint; run shadow vs production traffic.
  • Promote: on evidence and approval, flip production to the candidate version.
  • Operate: monitor drift/quality; rollback = repoint registry pointer and redeploy pointer.

Registry stages with gate in between.

Trained
Test gate
Staging
Approval
Production
Real world

In production

  • Standard releases: every production model change goes through the same gate pipeline.
  • Emergency rollback: a QoS regression reverts to yesterday's artifact instantly.
  • Multi-tenant: per-tenant model versions, all governed by one registry.
Guidelines

Best practices

  • Promote by artifact, not by notebook or git branch — the artifact is canonical.
  • Automate gates (metrics, schema, tests) but keep approval a human, logged decision.
  • Shadow EVERY candidate: diff scores vs production on real traffic before any flag flip.
  • Freeze production version; never mutate a promoted artifact.
Watch out

Common mistakes

  • Skipping staging 'because it worked in tests' and rolling out unshadowed.
  • Letting notebooks/pickles bypass the registry — ungoverned back doors.
  • Regression under traffic blamed on code when the registry pointer flipped silently.
  • Rollback that reintroduces the same bug because the 'older version' was the problem.
Root cause analysis

Common issues

Error
Half the fleet serves the new version
Root cause

Rollout not atomic or registry pointer read lazily

Fix

Read the version at startup, one pointer per deploy, and reconcile running pods.

Error
Promoted model regresses under traffic
Root cause

Shadow evidence weak or traffic composition changed

Fix

Gate promotions on shadow score diffs and pre-declared metrics; roll back immediately on breach.

Error
Model registry bypassed
Root cause

Engineers deploy from notebooks/pickles

Fix

Enforce the registry as the only deploy source via CI/ACLs and audit alerts.

When things break

Debugging tips

  • A brand-new regression on a freshly promoted model screams shadow-diff evidence wasn't read.
  • Verify the serving fleet reports the registry version; mismatched pods = partial rollout.
  • Test rollback in staging once per release, not just on incident day.
  • Audit registry events: promotion, shadow, rollback leave an event trail to correlate with incidents.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Rollback in a registry-based system usually means:

2

Shadow deployment changes user-facing behavior.

Go deeper

Official documentation

MLflow model registry
docs