MLOps Foundations
Intermediate 14 min read MLOps Foundations

MLOps Teams & Ways of Working

The model's lifecycle is a handoff problem as much as a tech problem

MLOps is also an org problem: who owns the model across its life, how data science, ML engineering and platform teams divide work, and how 'inner loop' experiments become 'outer loop' production systems.

team topologyrolesdata scientistML engineerhandoffinner loopouter loopplatform
Prerequisites: The ML Lifecycle
Before we start

What you will learn

  • Map core MLOps roles: DS, ML engineer, data engineer, platform
  • Describe the inner/outer loop and where handoffs fail
  • Choose ownership models that fit team size and model count
Introduction

What is this?

MLOps team workflows define who owns each stage of the lifecycle and how work moves between stages. The foundation is the two loops: the inner loop is fast experimentation (hypothesize, code, train, evaluate in a notebook); the outer loop turns winners into production systems (deploy, monitor, retrain). Teams choose an ownership model — embedded DS + ML engineers, a dedicated ML platform, or end-to-end 'ML engineers own everything' — and the choice shapes quality.

Motivation

Why do you need it?

  • Most failed ML projects are failed handoffs: notebooks thrown over the wall don't survive production.
  • Without clear ownership, drift and retraining have no owner and models quietly rot.
  • The inner/outer loop split is where reproducibility and ops discipline actually get built.
  • Ergonomics matter: the platform exists to shrink the inner loop latency, not to annoy researchers.
Think about it

Mental model

Imagine

Movie production: director, cinematographer, editor, crew

Think of

An ML product team

The director (data scientist) has the vision and iterates scenes (inner loop); the editor and crew (ML/platform engineers) transform the raw shoots into a shippable film (outer loop) with schedules and quality gates. Cast the wrong roles or blur the handoff and the premiere (launch) collapses.

Mechanics

How it works

  • Inner loop: data scientist/ML engineer iterates quickly — small data, one GPU, experiment tracking, fast feedback.
  • Outer loop: a winner is promoted through CI/CD, registry, staging and monitoring — owned by ML/product engineering.
  • Roles: DS owns problem quality; ML engineer owns system reliability; data engineer owns data paths; platform owns shared infra.
  • Handoff rituals: artifact + metadata + runnable pipeline, not a Jupyter notebook. A 'done' candidate comes with a runbook.
  • Platform teams instrument what ML teams consume: tracking, registry, compute queues — with SLOs.

Two loops and their owners.

Inner loop
Promote
Outer loop
Platform
Product
Real world

In production

  • A 5-person team: 2 DS in the inner loop, 1 ML engineer for pipelines, 1 data engineer, 1 platform engineer — one product owner per model.
  • A platform guild publishes templates for training, deployment and monitoring so DS rarely touches raw infra.
  • Monthly 'model health review' where drift alerts, retrains and SLOs are reviewed per model, owned end-to-end.
Guidelines

Best practices

  • End ownership at the model level: one named owner for quality AND operations.
  • Make the handoff an artifact with lineage and a runnable pipeline — notebooks are not handoffs.
  • Give the inner loop fast, safe guardrails (sandboxes, quotas) instead of friction that forces BYO-infra.
  • Write a runbook before the first deployment, not after the first incident.
  • Measure what matters: time from experiment to production, retrain success rate, and time-to-detect drift.
Watch out

Common mistakes

  • DS throwing a notebook over the wall 'for the engineers to fix' — no recoverable handoff.
  • A platform built by engineers that DS refuses to use, so shadow ops grows its own infra.
  • No owner for the monitoring-to-retraining loop; drift alerts page nobody in particular.
  • Rewarding experimentation only, while production maintenance is invisible and unrewarded.
Root cause analysis

Common issues

Error
Research invents its own infra
Root cause

Platform ergonomics/SLOs too weak for the inner loop

Fix

Instrument the platform with fast templates, quotas and clear SLAs; make the official path the path of least resistance.

Error
Models in production with no named owner
Root cause

Ownership ended at deployment; no ops owner assigned

Fix

Assign product + technical owners per model with a documented OLA and retrain trigger.

Error
Handoff mystery: notebooks can't reproduce the shipped model
Root cause

No formal artifact handoff contract

Fix

Require registry artifact + metadata + CI pipeline as the only accepted handoff.

When things break

Debugging tips

  • When models rot, check the org chart: who elected to own retraining? No owner = silent decay.
  • If the inner loop is slow, measure queue/utilization and environment setup time, not model speed.
  • When handoffs break, the artifact is usually missing metadata or a non-runnable pipeline — fix the handoff contract.
  • Use incident blameless reviews to find process gaps in ownership across stages.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which step most reduces 'notebook thrown over the wall' failures?

2

The monitoring-to-retraining loop should have a named owner.

Go deeper

Official documentation

Team topologies for ML
docs