Model Deployment
Intermediate 15 min read Model Deployment

APIs & Endpoints

The contract between your model and the world

A model API covers more than predict: request/response contracts, input validation, error handling, timeouts, observability and version pins. REST is the default; gRPC gives low-latency typed streaming.

RESTgRPCschematimeouterror handlingobservabilityapi contract
Prerequisites: Model Serving Inference Optimization
Before we start

What you will learn

  • Design prediction endpoints with explicit schemas and validation
  • Handle errors, timeouts and retries responsibly
  • Add observability hooks per-request
Introduction

What is this?

A prediction endpoint is a contract: an input schema (JSON/Proto), validation rules, a response structure, and declared failure modes. Practical details decide reliability: validate input early (reject malformed batches with actionable errors), cap request size, enforce timeouts, and return errors that don't leak internals. Observability logs the model version, input shape, latency and a freshness signature per request. gRPC adds typed streaming for high-QPS/low-latency fleets.

Motivation

Why do you need it?

  • Most 'model failures' in production are contract failures — wrong shape, missing fields, huge payloads.
  • Clear errors turn a 3am pager into a one-line fix.
  • Version pins at the model level keep client drift recoverable.
  • Per-request observability is the raw material for drift and anomaly detection.
Think about it

Mental model

Imagine

A bank teller with a deposit slip

Think of

A well-specified prediction API

The slip says exactly what's needed, the bank checks it before the transaction, and if it's wrong they tell you what to fix — not 'unacceptable deposit'. Machine clients, like people, need forms, validation and clear rejections, and the teller logs every transaction for the auditor.

Mechanics

How it works

  • Define schema (pydantic/JSON-schema/proto), enumerate the feature names and dtypes.
  • Validate on entry; return 4xx with precise reasons, never a 500 for bad input.
  • Enforce max rows/payload per call; large work belongs in batch endpoints.
  • Wrap the model: load artifact from registry at startup, pin versions, no reload per request.
  • Expose /health, /predict (+ optional /batch), and /version; log spans for latency + version.

Request lifecycle with validation and observability.

Request
Validate
Load pinned model
Predict
Log span
Real world

In production

  • Internal ML platforms: one predict API per model family, unified auth and logging.
  • REST-first for polyglot clients; gRPC for internal high-QPS scoring fleets.
  • Batch endpoints (accept arrays, return arrays) for dashboard bulk loads.
Guidelines

Best practices

  • Return 4xx for client errors and 5xx for model/infra errors — machines route on this.
  • Validate with the schema, reject fast, and echo the exact field that failed.
  • Put a timeout on EVERY model call; a stuck inference must not hang the request.
  • Log the model version + input hash on every request for later attribution.
Watch out

Common mistakes

  • Letting the model raise raw exceptions into HTTP 500s without context.
  • No payload cap — one huge batch melts the worker and OOMs the pod.
  • Idempotency ignored on retries → double-charged/duplicate prediction side effects.
  • Versionless endpoints masking which model a client actually talked to.
Root cause analysis

Common issues

Error
Every bad row a 500
Root cause

Model exception path unhandled

Fix

Catch, classify (input vs model error), return 4xx/5xx accordingly, and log stack without internals.

Error
Timeout storm during peak
Root cause

Undersized workers + retry amplification

Fix

Bound request size, size workers to p99 load, add circuit breakers and jittered retries.

Error
Two deployments, different behavior
Root cause

Each server loads a different artifact version

Fix

Read the pinned version from the registry/config; expose /version and alert on skew.

When things break

Debugging tips

  • Replay production payloads against the endpoint with a harness — most p50 mysteries reproduce locally.
  • Watch p99 vs p50 separately; a small batch of huge payloads distorts means.
  • Log the top field-errors by count; that histogram names your integration bugs.
  • Shadow-deploy a new version and diff per-request scores vs the old one.
Interviews

Interview checkpoint

Practice

Hands-on lab

Stand Up a Predict Endpoint
35 min · intermediate
In progress
Objectives
  • Expose a serialized model behind a small Flask-style endpoint
  • Validate inputs and return proper error codes
  • Add per-request logging with a version pin

Stand Up a Predict Endpoint

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A schema violation from a client should return:

2

Timeouts on model calls are optional if the model is fast.

Go deeper

Official documentation

MLflow model serving
docs