Model Deployment
Beginner 13 min read Model Deployment

Model Deployment Modes

Getting the model to do its job, where it's needed

Deployment is how a trained model becomes a usable service: online HTTP, batch jobs, embedded or edge. Choosing the mode is a latency, cost and freshness tradeoff, not a popularity contest.

deploymentonline servingbatchembeddededgeserving
Prerequisites: Model Registry CI/CD for ML
Before we start

What you will learn

  • Distinguish online, batch, embedded and edge deployment
  • Match deployment mode to latency/cost/freshness needs
  • Identify what a serving stack must provide regardless of mode
Introduction

What is this?

Deployment makes a trained model reachable by consumers. Online serving answers requests live with low latency; batch inference scores large sets on a schedule; embedded and edge modes run the model inside the product or on devices. The choice balances latency, cost, data freshness and governance.

Motivation

Why do you need it?

  • A fraud model that answers in 300ms has very different hosting needs than a nightly scoring job.
  • The right mode saves real money: batch compute is cheaper, online serving buys capability.
  • Freshness demands (how new must the answer be) rule out some modes.
  • Deployment mode shapes everything downstream: monitoring, versioning, rollback.
Think about it

Mental model

Imagine

A restaurant kitchen's three paths to the table

Think of

Deployment modes

Dine-in is online serving (cooked to order, fast, pricier staff), banquet prep is batch (cook ahead for many), and grab-a-box is embedded/edge (food travels with the guest). Each kitchen serves the same menu differently by demand and distance.

Mechanics

How it works

  • Online: the model is loaded in a long-lived server holding state in memory; requests get a prediction in ms.
  • Batch: a scheduler runs the model over a dataset window, writing scored outputs to storage.
  • Embedded: the model ships inside the application (e.g. mobile app) and runs on-device.
  • Edge: the model runs near the data source (gateway, camera) to cut round-trips and preserve privacy.
  • All modes still share: artifact versioning, monitoring, and a promote/rollback path.

The four deployment modes and their consumers.

Trained artifact
Online server
Batch job
Embedded/edge
Consumers
Real world

In production

  • A payment gateway serves a fraud model online because response time is a business metric.
  • A marketing team scores the full customer base nightly in batch — no one waits for a prediction.
  • A phone keyboard embeds the model locally; updates ship with app releases.
Guidelines

Best practices

  • Choose mode from requirements (latency, cost, freshness), not momentum.
  • Keep the serving artifact and versioning identical across modes.
  • Monitor whichever mode you choose — batch has quality metrics too.
  • Plan rollback in the mode you pick, not just the first environment.
Watch out

Common mistakes

  • Running batch jobs on online serving infrastructure and paying the premium.
  • Choosing online serving when freshness isn't needed and latency isn't critical.
  • Embedding a model with no update/refresh channel.
  • Assuming 'deployed' means online — batch monitoring still matters.
When things break

Debugging tips

  • High serving costs for a once-a-day job — you likely chose the wrong mode.
  • Latency targets impossible on your stack — check mode fit before tuning servers.
  • Model updates not reaching users — embedded/edge needs a rollout path, not just a redeploy.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What decides deployment mode more than anything?

2

Batch inference never needs model monitoring.

Go deeper

Official documentation

MLflow model serving docs
docs