Model Serving
Beginner 11 min read Model Serving

Serving Strategies

Turning weights into endpoints

Serving is how a trained model becomes an endpoint: choose hosted APIs, self-hosted inference servers, or edge deployables, then decide on batching, streaming and scales. Most teams start managed and graduate to self-hosted only for latency, data or cost reasons.

model servinginference serverself-hostedmanaged APIdeployment
Prerequisites: What Language Models Are
Before we start

What you will learn

  • Compare managed APIs vs self-hosted inference
  • Decide what drives placement and scaling
  • Define the pieces of a production serving stack
Introduction

What is this?

Serving is the deployment layer that exposes model weights as a stable API: an HTTP server loads a model, runs a forward pass per request, and returns generated tokens. The spectrum runs from fully managed provider endpoints (off-the-shelf, zero ops) to self-hosted GPU servers you control end to end.

Motivation

Why do you need it?

  • The same model behaves differently on different serving stacks — latency, throughput and cost vary.
  • Managed APIs trade control for speed to production; self-hosting trades ops for sovereignty and unit cost.
  • Serving decisions compound: they set your latency budget, failure surface and cost line.
Think about it

Mental model

Imagine

Serving

Think of

A restaurant kitchen

Same recipe, different kitchen: prep (batching), plating (streaming), and a host (load balancing) decide how many dishes go out per minute.

Mechanics

How it works

  • A serving process loads weights into GPU memory and exposes a predict/generate endpoint.
  • Requests may be batched to amortize GPU utilization — the main throughput lever.
  • Token streams return as they are produced; non-streamed calls wait for the whole completion.
  • An orchestrator (K8s, VM autoscalers, provider fleet) scales replicas against demand.

Serving stack in four layers.

Client
Gateway
Runtime
Model ID
Hands-on

Examples

Two ways to serve the same model

Two ways to serve the same model
// Managed: one HTTP call, provider does the rest
POST https://api.provider.dev/v1/chat/completions
{ "model": "fast-model", "messages": [{ "role": "user", "content": "hi" }] }
 
// Self-hosted: your own inference server
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192

Managed takes minutes and zero ops; self-hosted needs a GPU box, model download, and maintenance but keeps data on-prem and cuts per-token markup.

Real world

In production

  • A startup ships in a week on a managed API, then moves high-volume paths to self-hosted vLLM.
  • A bank self-hosts to keep prompts inside its network perimeter.
  • A platform team runs a gateway that fans out to managed and self-hosted replicas by cost tier.
Guidelines

Best practices

  • Start on a managed endpoint; introduce self-hosting behind an abstraction.
  • Decouple the model id from the client contract so you can swap engines.
  • Give every replica health checks and a readiness gate before it receives traffic.
  • Measure TTFT and tokens/sec before and after any serving change.
Watch out

Common mistakes

  • Self-hosting a model the traffic doesn't justify, taking on GPU ops for nothing.
  • Pointing clients directly at replicas, so a scale-out breaks every URL.
  • Choosing a model size you can't cheaply serve (peak GPU memory).
When things break

Debugging tips

  • TTFT spikes are usually queueing or cold cold-start, not model latency.
  • If tokens/sec drops when traffic rises, you are batch-starved or memory-bound.
  • Compare provider vs self-host latency with the same prompt and temperature.
Interviews

Interview checkpoint

Practice

Hands-on lab

Probe a Serving Endpoint
25 min · beginner
In progress
Objectives
  • Call a local mock 'inference' endpoint with curl
  • Read TTFT and tokens-per-second from the response
  • Compare streamed vs buffered delivery

Probe a Serving Endpoint

25 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is the main throughput lever on a self-hosted GPU?

2

Managed APIs always give the lowest cost per token.

Go deeper

Official documentation

OpenAI chat completions API
docs
vLLM architecture
docs