Production ML Architecture
Beginner 13 min read Production ML Architecture

Production ML Architecture

Process is not a place — architecture gives it one

A production ML platform is the set of systems that move data to models and predictions back to users — repeatedly, observably and safely. Architecture is where lifecycle process becomes a place with named parts and owners.

ML platformproduction MLarchitecturetraining platformserving platformML infrastructure
Prerequisites: The ML Lifecycle
Before we start

What you will learn

  • Name the planes of a production ML platform and what each is responsible for
  • Trace a prediction request through a complete platform
  • Separate concerns so a change in one plane does not cascade into another
Introduction

What is this?

A production ML architecture organizes the lifecycle into distinct planes: data, experimentation, training, registry, serving, monitoring and governance. Each plane is a set of services and owners with interfaces between them, so the platform can be built by many teams, scaled independently and operated reliably.

Motivation

Why do you need it?

  • A platform gives the lifecycle a home: 'where does this run' and 'who owns it' become answerable questions.
  • Separate planes let compute grow independently — training needs big GPU bursts, serving needs steady low-latency CPU.
  • Well-defined interfaces (model artifacts, versioned datasets, prediction APIs) are what make automation and governance possible.
  • Without architecture, every model ships its own ad-hoc stack and the team maintains N bespoke pipelines instead of one platform.
Think about it

Mental model

Imagine

A city's infrastructure

Think of

A production ML platform

Water, power, transport and sanitation are separate networks with separate owners that meet at standard interfaces — a building connects to all four. A model is a building; the platform is the grid it plugs into.

Mechanics

How it works

  • The data plane ingests, validates and versions datasets and labels, keeping raw and curated copies immutable.
  • The experimentation plane gives data scientists tracked runs, comparable metrics and shared compute.
  • The training plane executes reproducible training pipelines, sweeps and distributed jobs on elastic compute.
  • The registry plane stores model artifacts with metadata, signatures, lineage and promotion stages.
  • The serving plane exposes predictions via batch or real-time APIs, pulling features at request or compute time.
  • The monitoring and governance planes observe live behavior, detect drift and enforce approvals and audit trails.

The planes of a production ML platform.

Data plane
Experiment plane
Training plane
Registry plane
Serving plane
Monitoring plane
Governance plane
Real world

In production

  • A bank's platform registers every model in a central registry, so risk auditors get a complete lineage without talking to each data scientist.
  • An ads platform separates training (GPU spot burst nightly) from serving (auto-scaled CPU pool), so a training spike never degrades live latency.
  • A retailer's feature store serves the same feature values to both the nightly batch scoring job and the website's real-time API — one definition, two serving paths.
Guidelines

Best practices

  • Define the interfaces before the implementations: dataset version, model artifact format, prediction API schema.
  • Give every plane an owner — a platform you cannot page anyone for will fail silently.
  • Build shared primitives (registry, feature store, tracking server) once and reuse them; resist per-team copies.
  • Keep the platform observable: each plane exposes metrics, logs and traces at its boundary.
Watch out

Common mistakes

  • Buying a tool per plane with no integration, so lineage dies at the seams and governance is manual.
  • Building a platform for peak training load that sits idle 90% of the time and costs a fortune.
  • Treating the notebook as the deployment unit — notebooks wrap code, data and runtime together and are not production boundaries.
  • Designing the serving plane last, then discovering the model can't meet latency with the chosen feature path.
Root cause analysis

Common issues

Error
A new model can't be served because no one knows how it was packaged
Root cause

Training produced an artifact but the serving plane format was never defined

Fix

Standardize the artifact contract (framework, signature, dependencies) at the registry and gate registration on it.

Error
Retraining a model is blocked by a missing dataset
Root cause

Data plane has no retention/version policy; training data was overwritten

Fix

Version datasets as immutable artifacts and require a named dataset version for every training run.

When things break

Debugging tips

  • Trace a prediction through every plane to find where time or correctness is lost — most failures are interface failures.
  • Start at the data plane: verify the dataset the model was trained on still exists and is immutable.
  • Check plane boundaries for schema drift — the most common break is a handoff agreeing on nothing.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which plane stores model artifacts with lineage and promotion stages?

2

A notebook can serve as a production deployment boundary.

Go deeper

Official documentation

Google MLOps whitepaper — architecture & automation
docs
Microsoft — ML architecture & best practices
docs