Production ML Architecture
Intermediate 15 min read Production ML Architecture

Platform Blueprints

Three shapes that cover most of production ML

Most ML platforms are one of a few recognizable shapes: batch-scored, real-time scoring, or LLM-oriented. Choosing the blueprint up front sets the whole architecture, including cost and latency economics.

ML platformreference architecturebatch scoringreal-time scoringLLM platformblueprint
Prerequisites: Production ML Architecture
Before we start

What you will learn

  • Match a business need to a platform blueprint
  • Contrast batch-scored, real-time and LLM-oriented platforms
  • Price the latency/cost trade-offs each blueprint makes
Introduction

What is this?

A platform blueprint is a reference architecture you can start from instead of inventing a topology per project. The common shapes are the batch-scored platform (predictions computed on a schedule into a store), the real-time scoring platform (predictions computed per request at the API), and the LLM platform (foundation model + retrieval + guardrails).

Motivation

Why do you need it?

  • Blueprints encode hard-won defaults — the expensive lesson is learning which topology fits which problem.
  • A batch blueprint is dramatically cheaper and simpler; the data usually already lands on a schedule.
  • Real-time scoring changes every downstream contract: feature fetching, latency budgets, scaling.
  • Blueprints give teams a shared vocabulary and let you assess cost, latency and reliability before building.
Think about it

Mental model

Imagine

House plans an architect keeps

Think of

Platform blueprints

An architect doesn't redesign a house from nothing each time — ranch, split-level, modern — each plan has known costs and properties. ML platforms have the same few proven shapes.

Mechanics

How it works

  • The batch blueprint runs a scoring job on a schedule: latest features are joined, the model scores them, results land in a database for dashboards, reports or downstream systems.
  • The real-time blueprint serves a single prediction at request time: an API fetches fresh features (often from a feature store), runs the model in-process or via an inference server, and returns a prediction within a latency budget.
  • The LLM blueprint wraps a foundation model endpoint with prompt/retrieval pipelines, context stores and lightweight finetuned models for routing or guardrails — the heavyweight model is replaced by a managed API.
  • Hybrid blueprints score most traffic in batch and reserve the real-time path for the long tail of urgent decisions.
  • Every blueprint still shares the same registry, dataset versioning and monitoring planes — only the serving geometry changes.

Three platform blueprints and how predictions reach consumers.

Shared planes
Batch scoring
Real-time scoring
LLM serving
Consumers
Hands-on

Examples

A blueprint decision in one table

A blueprint decision in one table
# batch vs real-time vs LLM platform
def pick_blueprint(latency_ms, traffic, model_type):
    if model_type == "llm" or "foundation" in model_type:
        return "llm-platform: managed endpoint + retrieval + guardrails"
    if latency_ms and latency_ms < 300:
        return "realtime-platform: model server + online feature store"
    return "batch-platform: scheduled scoring into a prediction store"
 
for case in [(None, "daily", "xgboost"), (150, "high", "xgboost"), (None, "weekly", "gpt")]:
    print(case, "->", pick_blueprint(*case))

The blueprint choice falls out of three questions: how fresh must the prediction be, how much traffic, and what kind of model. None of the three needs a custom architecture.

Real world

In production

  • An insurance company scores all renewal likelihoods nightly in batch and writes results to Snowflake for sales workflows — zero serving infrastructure, sub-cent cost per prediction.
  • A rideshare app serves ETA predictions in real time from a Triton pool, with features read from an online store backed by Redis.
  • A support-automation startup uses an LLM blueprint: a managed chat model with retrieval over the KB and a guardrail classifier for disallowed topics.
Guidelines

Best practices

  • Prefer the batch blueprint unless the decision truly needs seconds-fresh predictions.
  • When real time is needed, overprovision for p99 latency, not just average — tail latency is the serving bugbear.
  • Keep the shared planes identical across blueprints so teams can shift shapes later.
  • Write a one-page blueprint note for each new platform: shape, latency budget, cost model, owners.
Watch out

Common mistakes

  • Building a real-time platform for a decision made once a day — wrong blueprint, ten times the cost.
  • Copying a blueprint element (e.g. 'feature store') without the problem it solves, adding moving parts for their own sake.
  • Assuming the LLM blueprint removes all serving concerns — retrieval quality, cost per call and guardrails are new problems, not no problems.
Root cause analysis

Common issues

Error
Real-time predictions cost 50x the batch equivalent
Root cause

The decision doesn't need sub-second freshness but got a real-time blueprint

Fix

Move to scheduled batch scoring and keep a thin real-time path only for genuinely urgent edge cases.

Error
LLM inference bill doubles every month
Root cause

No caching or routing; every request hits the largest model with long prompts

Fix

Add response caching, cheaper models for easy cases, and prompt truncation/retrieval to bound context length.

When things break

Debugging tips

  • Re-check the freshness need when a blueprint feels over-engineered; a batch job may be the honest fix.
  • For real-time platforms, profile where the milliseconds go: feature fetch, model exec or serialization.
  • For LLM platforms, measure tokens and feedback loops before touching model choice — cost usually lives there.
Interviews

Interview checkpoint

Practice

Hands-on lab

Price Three Platform Blueprints
30 min · intermediate
In progress
Objectives
  • Model the cost and latency of batch, real-time and LLM blueprints from simple assumptions
  • Compare total hourly cost across traffic levels
  • Pick the right blueprint for a scenario and justify it with numbers

Price Three Platform Blueprints

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which blueprint fits 'renew each user's support-ticket priority instantly when they open the app'?

2

Every ML platform needs a separate, custom architecture before it can be production-grade.

Go deeper

Official documentation

Google Cloud — ML architecture center
docs
AWS — ML best practices (architecture)
docs