Model Serving Patterns
Batch, online, streaming: pick the contract first
Serving is how a model answers real calls: batch jobs for bulk scoring, online endpoints for low-latency requests, streaming for events. Choosing the pattern sets latency, cost and freshness expectations before any code.
What you will learn
- Compare batch vs online vs streaming serving
- Match patterns to latency/service-level requirements
- Design deployments that combine patterns safely
What is this?
Serving patterns describe how prediction requests reach a model. Batch: a scheduled job scores a large dataset (deltas, nightly scores) — high throughput, forgiving latency. Online: an API scores one request in ~ms-tens of ms (recommendation, risk). Streaming: events trigger predictions in near-real-time (fraud blocks). Real systems usually mix: an online model plus nightly batch refresh and a streaming path for anomaly spikes.
Why do you need it?
- The pattern determines cost per prediction, hardware, and even the model size you can afford.
- Latency SLOs gate architecture: 'must answer in 50ms' forbids batch-only.
- Freshness needs vary: fraud needs seconds, churn scores need overnight.
- Choosing right up front avoids ripping out an endpoint for a batch pipeline mid-project.
Mental model
A bakery vs a pre-made delivery route
Online vs batch serving
Walk-in customers demand a pastry in minutes (online, low-latency, expensive per item). Route delivery fills thousands of scheduled boxes overnight (batch, cheap per item, latency-tolerant). A deli that starts baking after orders arrive (late binding) never hits either contract.
How it works
- Online: request → feature transform → model forward → response; one per call, latency bound, high availability needed.
- Batch: cron/spark job pulls a dataset, scores rows in parallel, writes results to a table/object store.
- Streaming: consumer reads events, scores in-process, emits decisions downstream (fraud, abuse).
- Hybrid: online API for interactive paths + scheduled batch for precached/dashboards.
- Contract: define SLA (p50/p95 latency, successful rate), throughput, and staleness budget per pattern.
Three serving lanes into one model.
In production
- Recommendation: online rec API + nightly batch precomputation for cold loads.
- Risk: online fraud scores; batch risk portfolios each morning.
- Marketing: weekly batch propensity scores pushed to campaign tools.
Best practices
- Write the SLA first (latency p95, QPS, freshness) and let it pick the pattern.
- Keep online paths stateless and horizontally scalable; move heavy scoring off the request path.
- Make feature computation shared between batch and online to avoid train/serve skew.
- Precompute what rarely changes; re-score only what changed (incremental).
Common mistakes
- Building an online endpoint for a workload that was always batch-worthy (cost + p99 flakiness).
- Serving a model online with batch-only feature pipelines — every request stalls.
- Batch jobs that score stale features but claim freshness.
- No backpressure/dead-letter handling on streaming deciders.
Common issues
Online p95 blows the SLAHeavy feature fetch on the request path
Cache features, precompute, or move that path to streaming/batch.
Online and batch scores disagreeFeature pipeline/timing skew between the two paths
Share one feature library and log feature drift between paths.
Streaming consumer falls behindModel call serial per event
Batch/scalar micro-batching, parallelize consumers, and size the DLQ for retry storms.
Debugging tips
- Profile the request path: feature-transform time vs forward time often surprises.
- Load-test batch jobs at peak dataset size; queue time is the sneaky latency.
- For streaming, log event timestamps end-to-end to catch processing lags.
- Compare batch vs online scores on an overlap sample — skew detection built in.
Interview checkpoint
Quick quiz
A real-time fraud-blocking decision best maps to:
Mixing batch and online paths for one model is usually a mistake.