Online Servers and Engines
The engine under the API surface
The serving engine is the performance heart of real-time inference: resident models, batching, GPU scheduling, and optimized runtimes. Choosing and configuring it — TorchServe, Triton, vLLM, engineered REST — decides whether latency budgets actually hold.
What you will learn
- Map engine responsibilities: residency, batching, scheduling
- Choose an engine/runtime for model type and workload
- Configure batching and worker pools deliberately
What is this?
An online serving engine is the runtime that keeps models resident, schedules inference on CPUs/GPUs, batches requests and serves contracts (HTTP/GRPC) at performance. TorchServe and Triton are general engines; vLLM specializes LLM serving; engineered REST sits on top for control.
Why do you need it?
- A resident model beat a per-request reload by orders of magnitude in latency.
- Batching at the engine level boosts GPU throughput without client coordination.
- Engine-managed workers, scheduling and health free you from re-implementing them.
- The right runtime per model class (optimized kernels, KV caching) is a large latency lever.
Mental model
A courier hub with sorting lanes and press-out schedulers
Serving engine
Packages (requests) arrive in bursts. The hub (engine) sorts into lanes (queues), packs trucks for efficiency (batching), and dispatches on a schedule (scheduling) — each lane's efficiency decides whether the same-day deadline (latency SLO) is met.
How it works
- Models load once into memory; workers serve requests against them.
- Requests queue; dynamic batching packs pending work into efficient forwards.
- The scheduler multiplexes models/GPUs respecting concurrency limits.
- Optimized runtimes apply kernels (TensorRT, ONNX) or LLM-specific features (continuous batching, KV cache).
- Health, versioning and metrics endpoints keep the deployment observable.
Engine internals under the API.
In production
- A Triton instance serves three fused models with dynamic batching at p95 < 30ms.
- An LLM API uses vLLM with continuous batching and KV caching to raise token throughput.
- A small team uses a hand-rolled FastAPI + resident ONNX model for a simple single-model workload.
Best practices
- Match engine to model class — don't haul vLLM for a tabular model, or a bare REST server for tensors.
- Enable and tune batching; measure p95 before and after with realistic traffic.
- Set concurrency limits so one model can't starve the scheduler.
- Warm models and preload cache before accepting traffic.
Common mistakes
- Using an optimized engine but disabling batching, leaving utilization on the table.
- One worker per model request — through-the-roof serialization.
- Treating the engine as a black box, never reading its health/config telemetry.
- Runtime image drift between environments with no pinning.
Debugging tips
- Latency plateau — inspect scheduler and batch-window settings, not just the model.
- GPU utilization low — requests may batch poorly; check arrival distribution vs batch window.
- Memory grows — unloaded versions or leaked cache; check model lifecycle controls.
Interview checkpoint
Hands-on lab
- Model batching windows and their latency effects
- Measure end-of-window flush impact
- Choose window size from the p95 constraint
Tune Batching for Latency
35 min · intermediate
Quick quiz
The serving engine's main throughput lever is…
Every model should be served by the newest LLM engine.