Production ML
Advanced 16 min read Production ML

Production ML Architecture

The model is one node in a system you operate

A production ML system is a connected architecture — data sources, feature pipelines, model server, orchestration, monitoring, retraining — designed around SLAs, failure isolation and graceful degradation, not just the model.

architecturesystem designfeature pipelinemodel serverorchestrationSLAdegradation
Prerequisites: Model Serving What is MLOps?
Before we start

What you will learn

  • Design the end-to-end architecture of a production ML system
  • Place features, model server, monitoring and retraining with the right contracts
  • Design for failure: isolation, queues, fallbacks and degradation
Introduction

What is this?

Production ML architecture is the design of the whole system around the model: how features get computed and served, how the scoring endpoint scales, how scheduled jobs and online requests coexist, how monitoring feeds retraining, and how the system degrades when any part fails. The model is central but small — the architecture decides latency, freshness, cost and whether a failure is an incident or a non-event.

Motivation

Why do you need it?

  • Real systems fail in the seams: the model is fine, but features are stale, the endpoint throttles, or monitoring stops.
  • SLAs force architecture choices: a 20ms prediction limit forbids 'compute features from 4 databases on the fly'.
  • Isolation and queues turn partial failures into degraded-but-available service.
  • A designed feedback loop separates 'systems that run' from 'systems that improve'.
Think about it

Mental model

Imagine

An airport's architecture

Think of

A production ML system

Terminals (online serving), cargo schedules (batch), and air traffic control (orchestration) are separate systems with different tolerances. When the cargo runway is fogged in, passengers still fly — because each lane fails independently and the whole system degrades gracefully instead of all at once.

Mechanics

How it works

  • Data layer: raw sources feed batch (warehouse/lake) and streaming (Kafka) paths.
  • Feature layer: a feature store/serving layer provides consistent features for training and inference.
  • Model layer: versioned artifacts served by a low-latency API (online) or a scheduled batch scorer.
  • Orchestration layer: schedules training, backfills, batch scoring; retries and alerts.
  • Feedback layer: logs predictions plus outcomes, computes drift/proxy metrics, feeds retraining triggers.

The pieces of a production ML system.

Sources
Feature pipelines
Model server
Consumers
Monitoring
Retraining
Real world

In production

  • A bank runs fraud scoring as an online API backed by streaming features, with a nightly batch scorer for portfolio re-scoring and a monitoring loop that pages on score distribution shifts.
  • A retailer's recommendation stack keeps the model API stateless and horizontally scalable, while a nightly job precomputes cold-start scores; the system falls back to 'popular items' when the API errors.
  • A logistics company serves ETAs with feature fallbacks: when a GPS feature lags, a cached/static estimate is served so the ETA never goes missing.
Guidelines

Best practices

  • Write SLAs per lane (online latency p95, batch freshness) before wiring the system.
  • Make fallbacks explicit: every critical online path needs a defined degraded mode.
  • Keep the online path stateless and idempotent; move heavy recomputation off the request path.
  • Log prediction requests at the boundary with feature snapshots for debugging and drift analysis.
  • Design retraining as a first-class pipeline with gates, never a manual notebook on a cron.
Watch out

Common mistakes

  • Designing around the model and forgetting the feature path until the first latency incident.
  • A single shared 'model service' for everything, coupling a bursty recommender to a critical fraud endpoint.
  • No fallback: one feature source outage takes the whole prediction down.
  • Batch and online architectures that don't share feature code, guaranteeing drift-by-construction.
Root cause analysis

Common issues

Error
Online API latency spikes at business hours
Root cause

Feature recomputation on the request path scales linearly with traffic

Fix

Precompute and cache features; move heavy joins to batch and serve them from the online store.

Error
One model's surge takes down the shared service
Root cause

No isolation or rate limiting between workloads on the same endpoint

Fix

Split services by SLA, add per-tenant rate limits and queue/backpressure gates.

Error
Predictions go missing during a feature outage
Root cause

No fallback model or static-feature path

Fix

Define and exercise a fallback (cache, static estimates, popular content) before the outage.

When things break

Debugging tips

  • Trace one request end-to-end: feature fetch, model inference, response — the latency budget reveals weak spots.
  • Load-test each lane separately; a batch job and an online API have different failure modes.
  • When a prediction is wrong, pull the logged feature snapshot — the model didn't see what you think it saw.
  • Simulate outages (kill the feature store, kill Kafka) in staging and watch degradation kick in.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which failure is most likely to take down an online ML system even when the model is fine?

2

Every critical online prediction path should have an explicit degraded/fallback mode.

Go deeper

Official documentation

Rules of ML (Google)
docs