CI/CD for ML
Advanced 14 min read CI/CD for ML

Pipeline Gates and Approvals

Decision points that enforce what teams mean

Gates are automated decision points between pipeline stages that turn 'we think' into enforced checks. Approvals add human judgment where automated evidence can't yet decide — a calibration between risk and speed.

pipeline gateapprovalchampion thresholdrelease policygo/no-go
Prerequisites: CD for ML Model Monitoring
Before we start

What you will learn

  • Design gates at stage boundaries with explicit criteria
  • Distinguish automated gates from human approvals
  • Tune gate strictness to cost-of-wrong vs cost-of-delay
Introduction

What is this?

A gate is a checkpoint between pipeline stages that evaluates explicit criteria — metrics against champion, data quality, compliance flags, manual approval — and blocks passage until they pass. Gates convert a team's risk policy into enforced, audited behavior.

Motivation

Why do you need it?

  • Without gates, promotion is a habit, not a policy.
  • Automated gates enforce at 3am what a cautious human would only do awake.
  • Manual approvals reserve human judgment for judgment-worthy calls.
  • Gates yield an audit trail: who/what, when, with what evidence.
Think about it

Mental model

Imagine

A nightclub's bouncer at the door plus an owner at VIP

Think of

Pipeline gates + approvals

The bouncer (automated gate) checks ID (evidence) against clear rules at max enforcement. The owner (human approval) handles judgment calls and edge guests. Neither substitutes for the other.

Mechanics

How it works

  • Define gate criteria at each boundary: 'AUROC ≥ champion, drift in range, no blocking compliance flags'.
  • The pipeline runs evidence collection before the gate: eval, telemetry, vulnerability scan.
  • The gate evaluates criteria and either passes, blocks (with reasons), or queues for human.
  • Approval tools (registry approvals, PR reviews, workflow approvals) record the decision.
  • Denied passage leaves the artifact at its stage with an auditable veto record.

Gate placement between delivery stages.

CI result
Quality gate
Approval gate
Stage promote
Audit log
Real world

In production

  • A champion-taught gate blocks any promotion whose candidate doesn't beat the production AUROC.
  • A financial model requires a human approval gate at production in addition to automated checks.
  • A vulnerability scan gate stops a model whose serving image contains a critical CVE.
Guidelines

Best practices

  • Make criteria explicit and quantitative before the release, not during it.
  • Put automated gates everywhere; put humans only where judgment adds value.
  • Record pass/veto with the exact evidence snapshot for audit.
  • Revise thresholds when the champion itself changes — stale gates promote mediocrity.
Watch out

Common mistakes

  • Gates with vague criteria ('looks good') that pass everything.
  • Human approval added to trivial changes, slowing flow with no added safety.
  • Gate thresholds not compared against the champion, so relative regressions sail through.
  • No audit trail — the gate that 'approved' can't be reconstructed later.
When things break

Debugging tips

  • Gate unexpectedly blocking — print the criteria values vs thresholds, don't guess from the summary.
  • Gate approving everything — the comparator may be stuck comparing to an old champion; verify the champion ref.
  • 'Approved but wrong' — check who/what approved and whether the evidence was current at decision time.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Where should human approval gates be sprinkled?

2

Every gate decision should be recorded with its evidence snapshot.

Go deeper

Official documentation

MLflow model registries adoption
docs