Production ML
Advanced 15 min read Production ML

Cost, Governance & Compliance

Models that bend budgets and legal budgets alike

Production ML spends real money and carries real obligations: GPU cost controls, data lineage, auditability, model documentation, bias and privacy compliance. Governing those constraints is as important as the accuracy metrics.

costGPU spendgovernancecompliancelineageauditmodel documentationbiasprivacy
Prerequisites: Model Registry Production ML Architecture
Before we start

What you will learn

  • Control ML cost across training, serving and experimentation
  • Build lineage, audit trails and model documentation for compliance
  • Address bias, privacy and regulatory obligations in model design and operations
Introduction

What is this?

Cost and governance treat ML as a portfolio to budget, own and audit. Cost covers training (GPU utilization, spot strategy, quotas) and serving (batch vs online, caching, autoscaling). Governance covers data lineage (what raw data fed which model), audit trails (who promoted what when, on what evidence), model documentation and risk assessment, and compliance with privacy (PII minimization, retention), fairness (bias audits) and domain regulation (healthcare, finance). A production model without lineage and documentation is a compliance incident waiting to happen.

Motivation

Why do you need it?

  • GPU spend is the first thing finance notices; uncontrolled experimentation burns budgets monthly.
  • Regulators ask 'what does this model do and on what evidence' — registry lineage answers before the auditor does.
  • Bias found late is expensive: post-hoc mitigation can't fully repair a biased data pipeline.
  • Compliance frameworks (EU AI Act, GDPR) increasingly require documentation and human-oversight artifacts.
Think about it

Mental model

Imagine

A business division's P&L and audit committee

Think of

ML cost governance

Every initiative budgets (cost), every contract is traceable (lineage), every decision is minuted (audit trail), and the committee can answer for any one expense or decision. A model stack with no ledger is a division with no finance function.

Mechanics

How it works

  • Cost: tag every run by team/model; set quotas and budgets; use spot/preemptible for preemptible jobs; cache and autoscale serving; separate experimental vs production spend.
  • Lineage: record data versions, code hashes, hyperparameters and artifacts into the registry for every model.
  • Audit: promotions, shadow runs, rollbacks and approvals carry event logs with actors and evidence.
  • Documentation: model cards/risk sheets encoding purpose, data, metrics, fairness and decisions.
  • Compliance: PII minimization in features, retention limits, explainability hooks and human-review mechanisms for high-risk decisions.

Cost and governance wrap the whole lifecycle.

Budget & tag
Train
Lineage & audit
Document
Operate
Real world

In production

  • An ML platform team tags every training job with team and budget code; quotas stop a runaway search from exhausting the month.
  • A bank maintains full lineage per credit model: training data versions, champion challenger evidence, auditor-queryable by model ID.
  • A healthcare ML provider writes model cards, runs fairness audits on protected attributes and keeps human-review pathways for high-risk outputs.
Guidelines

Best practices

  • Set spend visibility first, then quotas — nobody must guess what a run costs.
  • Automate lineage capture; manual metadata is always incomplete.
  • Write model cards as living documents updated at promotion time, not retro archives.
  • Run bias audits per protected attribute BEFORE high-stakes deployment, and re-run them at retrain.
  • Apply retention and privacy rules to logged prediction data as strictly as to source data.
Watch out

Common mistakes

  • Uncapped experimentation: a 'quick test' consumes the GPU budget a production retrain needed.
  • Model promotion without lineage — auditors and incident investigators both hit a wall.
  • Documentation as an afterthought: cards written post-hoc that never match the artifact.
  • Ignoring bias monitoring at retrain: the world changed and the fairness audit expired with the old weights.
Root cause analysis

Common issues

Error
GPU spend tripled with no new projects
Root cause

Orphaned/reserved GPUs, retries-on-errors, and spot-vs-on-demand drift

Fix

Reconcile allocations, label and enforce quotas, and auto-release idle reserved capacity.

Error
Auditor asks 'what model is in production on what data' and nobody can answer cleanly
Root cause

No enforced lineage at the registry

Fix

Make the registry the canonical promotion path so lineage is captured by construction.

Error
Bias regression after a retrain
Root cause

The retrain used recent data whose cohort composition changed

Fix

Re-run fairness audits as a non-negotiable retrain gate, with cohort-level data checks early.

When things break

Debugging tips

  • When spend spikes, break cost down by job tag and idle-vs-active GPU: idle reserved GPUs are usually the culprit.
  • For audit questions, start from the registry pointer: model ID → promotion log → training run → data version.
  • A bias finding is best debugged at the data layer: cohort-level label and feature skew, not the algorithm.
  • Timebox governance checks in CI — lineage, doc and fairness gates should run automatically with every promotion.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The fastest way to an auditor-ready lineage trail is:

2

Fairness audits only need to run once at the initial model release.

Go deeper

Official documentation

EU AI Act — risk & documentation
docs
Model cards for model reporting
docs