Production ML Architecture
Intermediate 13 min read Production ML Architecture

Platform Teams and Ownership

Own the planes, not just the pipelines

Architecture diagrams fail when nobody owns a plane. Production ML teams split along the same seam as the platform: data/platform/ML engineers each own one interface and page on its failures.

ML team structureplatform engineeringMLOps rolesownershiphandoffson-call
Prerequisites: Production ML Architecture
Before we start

What you will learn

  • Map platform planes to owning teams
  • Define clean handoffs at plane boundaries
  • Design on-call and paging for a platform, not just an app
Introduction

What is this?

Platform ownership assigns each plane of the ML architecture to a team with clear interfaces and paging responsibilities. Data engineers own the data plane, ML/platform engineers own training and serving, and an MLOps or SRE line owns the shared reliability. The core principle: the team that changes a plane owns its runtime behavior.

Motivation

Why do you need it?

  • Unowned planes drift: the registry nobody maintains stops being used, and lineage dies with it.
  • Clear interfaces let teams move fast without a daily architecture meeting.
  • On-call is impossible without ownership — you can't page 'the ML department'.
  • Ownership gaps are the top cause of silent ML incidents: a handoff both sides assumed the other covered.
Think about it

Mental model

Imagine

A borough's public utilities

Think of

Platform team ownership

Water, power, transit and sanitation each have an operator with a phone number. When your lights flicker, you call the power company, not 'the city'. Same for model planes.

Mechanics

How it works

  • A platform/ML-infrastructure team owns the shared primitives: registry, feature store, tracking server, serving runtime.
  • Data engineers own the data plane, including dataset versioning, validation gates and freshness SLAs.
  • Applied ML teams own their models' lifecycle within the platform, from experiment to promotion to retirement.
  • MLOps/SRE owns reliability across the planes: metrics, SLOs, on-call routing and incident response.
  • Every interface is documented with an owner and a paging path, and handoffs are tested — ownership means 'if it breaks at 3am, the on-call can rebuild it'.

Who owns which plane of the platform.

Data plane
Platform plane
Model plane
Reliability plane
Consumers
Real world

In production

  • A fintech's platform team maintains one model server image all teams deploy, so a grading latency fix ships once and everyone inherits it.
  • An e-commerce company runs an on-call rotation where the platform team pages on serving p99 breaches and the applied-ML team pages on metric regressions.
  • A healthcare org documents every handoff (dataset → training → registry → serving) with an owning team and a runbook, so an incident gets a point of contact in minutes.
Guidelines

Best practices

  • Write one ownership matrix: plane, owning team, interface, page path, runbook link.
  • Make the platform a product with a roadmap and a lightweight intake, not a ticket queue.
  • Require runbooks for every plane before it goes to production on-call.
  • Test handoffs quarterly — re-run a model from a registry artifact and prove the whole chain still works.
Watch out

Common mistakes

  • A 'shared services' team that owns everything, which becomes the bottleneck and the un-pageable catch-all.
  • Ownership by creation: whoever built it owns it forever, so old planes rot with no retirement path.
  • Writing ownership docs that nobody followed when the incident hit — runbooks must be exercised, not filed.
Root cause analysis

Common issues

Error
A model goes stale and nobody knows who refreshes it
Root cause

Applied-ML team assumed platform retrains; platform assumed the ML team does

Fix

Encode the retrain schedule and owner in the registry metadata so the responsible plane is unambiguous.

Error
On-call paged for a serving issue but no team had a runbook
Root cause

Plane shipped without operations ownership

Fix

Block production promotion until the owning team publishes and drills a runbook.

When things break

Debugging tips

  • When an incident spans planes, ask who owns the boundary — the gap is the bug.
  • Check the runbook before escalating; a paged team that has no runbook is a handoff failure.
  • Post-hoc, review which teams changed what near the incident — ownership drift usually accompanies interface drift.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which team most naturally owns the model registry and serving runtime?

2

Ownership is only needed for things that changed recently.

Go deeper

Official documentation

SRE — on-call & operational practices
docs
Google — platform engineering overview
docs