Production ML Architecture
Intermediate 14 min read Production ML Architecture

Versioning and Governance Tier

The paper trail every production model needs

Versioning and governance are what let a platform be audited, rolled back and compliant. The registry is the hub: artifacts, lineage, approvals and every promotion recorded as an immutable event.

model governanceversioningauditregistrylineageapprovalscompliance
Prerequisites: The Model Registry
Before we start

What you will learn

  • Encode promotion workflows in the registry
  • Show how lineage links datasets, runs, code and models
  • Design audit trails that satisfy compliance review
Introduction

What is this?

The versioning and governance tier makes every model an auditable, reproducible object: its artifact, metadata, lineage to training data and code, approval state and deployment history all recorded centrally. Governance turns 'who approved this' and 'what would break if we roll back' from tribal knowledge into queryable facts.

Motivation

Why do you need it?

  • Compliance (SOC 2, HIPAA, GDPR, model-risk review) requires demonstrating controlled, reviewed model changes.
  • Rollback needs versioning: you can't revert to what you didn't store.
  • Automation needs gates: CI/CD can auto-promote only because the registry encodes what 'ready for production' means.
  • Lineage answers the incident question 'what trained this?' without a hunt across notebooks.
Think about it

Mental model

Imagine

An aircraft's maintenance log

Think of

Model versioning and governance

Every flight is tied to a maintained, signed, logged airframe; you fly because the log proves it. Production models fly the same way.

Mechanics

How it works

  • Training writes a versioned artifact plus metadata (metrics, data version, code commit, framework, signature) to the registry.
  • A promotion workflow moves the model through stages — staging → production — with required approvals and quality gates at each hop.
  • Every promotion, deployment and rollback is recorded as an immutable event with actor, time and artifact version.
  • Deployments reference a registry version and stage, so the serving tier and the audit log agree on what is live.
  • Governance reports — model inventory, lineage trees, promotion history — are generated from the same records.

Governance around the registry.

Training run
Register
Approve
Promote
Deploy
Audit
Real world

In production

  • A bank's model-risk review pulls a lineage report per model before any production promotion — the entire package is generated from registry records, not spreadsheets.
  • An e-commerce CI pipeline auto-promotes a model to production only when offline metrics pass and the data version is current; humans approve only the final rollout.
  • A medical-device vendor keeps every deployed version for 7 years; rollback is a one-command deploy of a registry-backed artifact.
Guidelines

Best practices

  • Make the registry the single source of truth for what is in production; serving pulls by version, never by ad-hoc file path.
  • Encode, don't narrate, the promotion policy: quality gates run, approvals are recorded, non-authorized paths fail.
  • Attach data version and code commit to every model so lineage is complete and rebuildable.
  • Immutable artifact store: never overwrite a version — rollback needs the old bits to still exist.
Watch out

Common mistakes

  • Versioning the model file but not its dataset or code — a 'versioned' model that can't be reproduced is a versioned lie.
  • Human approval as an email thread outside the registry, so the audit log and reality disagree.
  • Letting teams deploy arbitrary files alongside the registry, quietly bypassing governance.
Root cause analysis

Common issues

Error
Auditors find a production deployment with no approval record
Root cause

A team deployed outside the registry or the approval step was advisory

Fix

Make the registry the only deploy source and enforce the gate in the pipeline, not in a review doc.

Error
Rollback references a version that no longer exists
Root cause

GC removed old artifacts

Fix

Exclude served/approved versions from garbage collection; keep a retention policy for governed artifacts.

When things break

Debugging tips

  • In an incident, pull the lineage for the live version first — it names dataset, code and metrics in one query.
  • If a rollback fails, check that the previous artifact wasn't garbage-collected.
  • Diff the audit log against actual deployments regularly; divergence means a bypass path exists.
Interviews

Interview checkpoint

Practice

Hands-on lab

Govern a Model Lifecycle in a Registry
35 min · intermediate
In progress
Objectives
  • Register staged model versions with metadata
  • Simulate promotion gates and approval records
  • Produce an audit report from registry records

Govern a Model Lifecycle in a Registry

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which is the single source of truth for what model is live in production?

2

A versioned model artifact can be safely re-saved or overwritten later.

Go deeper

Official documentation