Model Deployment
Intermediate 15 min read Model Deployment

REST API Servers

A model file becomes a request/response service

The REST API server turns a model file into a request/response service: load the artifact, expose predict endpoints, batch requests, and return the contract alongside the prediction. Small and standard, it's how most models first meet their users.

rest apiprediction endpointflask/fastapirequest batchingserving contract
Prerequisites: Model Deployment Modes Model Artifact Serialization
Before we start

What you will learn

  • Serve a model behind a REST predict endpoint
  • Define a response contract with metadata
  • Enable request batching and error semantics
Introduction

What is this?

A REST API server loads a serialized model into memory and exposes HTTP endpoints — typically /predict — that parse requests, run inference, batch when able, and return the prediction with metadata. It's the default on-ramp for 'make the model callable'.

Motivation

Why do you need it?

  • REST is universal: any client library, language or service can call it.
  • A wall between model and consumer makes validation, versioning and A/B possible.
  • Batching at the server turns many small requests into fewer model calls.
  • Server-side load means the model isn't re-spun per request from scratch.
Think about it

Mental model

Imagine

A concierge at a hotel desk

Think of

REST prediction server

Guests (clients) don't talk to housekeeping directly; the concierge (API) takes requests, checks understanding (validation), groups same-floor errands (batching), and returns confirmations (metadata). One desk, many guests.

Mechanics

How it works

  • The server loads the artifact at boot and keeps it resident (one model load for all requests).
  • Requests hit /predict; the server validates input shape and types.
  • A worker pool runs inference; concurrent requests can be batched.
  • The response carries the prediction, model version and timing metadata.
  • Health endpoints (/ready, /live) keep orchestration honest.

A REST serving round trip.

Client
Validation
Batch
Inference
Response
Hands-on

Examples

A minimal FastAPI predict endpoint

A minimal FastAPI predict endpoint
from fastapi import FastAPI
from pydantic import BaseModel
import joblib
 
app = FastAPI()
model = joblib.load("model.joblib")
 
class Payload(BaseModel):
    features: list[float]
 
@app.post("/predict")
def predict(p: Payload):
    prob = model.predict_proba([p.features])[0][1]
    return {"prob": prob, "model_version": model.meta["version"]}

The model loads once into global state; each request is validated by Pydantic, scored, and answered with the prediction plus the served version.

Real world

In production

  • A scoring service answers ~50ms predictions behind a load balancer, one model load across thousands of calls.
  • A batch-aware server groups bursts of fraud checks into single model calls.
  • Versioned servers let a canary route to /predict-v2 during rollout.
Guidelines

Best practices

  • Return the model version and input checksum in every response.
  • Validate input server-side; don't trust clients or internal callers.
  • Expose /live and /ready for orchestrators to health-check.
  • Cap request sizes and timeouts; a prediction endpoint is a service, not a notebook cell.
Watch out

Common mistakes

  • Reloading the model on every request in the handler.
  • Returning bare floats with no version or metadata — undebuggable in production.
  • No timeouts, so a slow batch-of-requests occupies workers forever.
  • Trusting raw JSON and crashing on unexpected shapes instead of validating.
When things break

Debugging tips

  • 'Works in notebook, 500 in API' — the request shape differs; log validation failures verbatim.
  • Latency climbs with concurrency — check worker pool size and whether requests serialize on a lock.
  • Model version confusion — read the response's version field, not the manifest on disk.
Interviews

Interview checkpoint

Practice

Hands-on lab

Serve a Model Behind REST
35 min · intermediate
In progress
Objectives
  • Load a model once and serve /predict
  • Return versioned responses with timing
  • Exercise readiness and failure paths

Serve a Model Behind REST

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What should every predict response include beyond the prediction?

2

A prediction server should validate request payloads server-side.

Go deeper

Official documentation

FastAPI docs
docs
MLflow deploy models to REST
docs