Containerizing Models
One image = one reproducible deploy unit
Containers make a model deployable anywhere: code, deps, artifact and runtime in one reproducible image. Key arts: correct base image, small layers, GPU runtime support, health checks and artifact injection instead of baking.
What you will learn
- Structure a Dockerfile for a model service
- Serve GPU vs CPU images correctly
- Inject/inject-verify artifacts at deploy time
What is this?
Containerizing wraps the model service — code, dependencies, runtime and (often) the artifact — into an immutable image that runs identically on any host. Good practice: pin base images, keep layers small, split CPU/GPU images, run as non-root, add healthchecks and readiness, and either copy the artifact at build or mount it at deploy. Images register the exact environment as part of the deployable, killing 'works on my machine'.
Why do you need it?
- Reproducibility: the env is part of the artifact, not folklore.
- Rollouts and rollbacks become 'deploy image X, and Y to undo'.
- GPU model images capture the whole CUDA/cuDNN stack reliably.
- Horizontal scaling is identical replicas of one image — no config drift.
Mental model
A food truck that tows its own kitchen
A containerized model service
The truck carries the stove, ingredients (deps) and the menu (code/artifact); it parks anywhere and serves the exact same dish. Your model does the same: same Python, same framework, same weights, in one rolling kitchen that can spawn a thousand clones.
How it works
- Base image: python slim/prod base, PINNED (not :latest); add GPU base for CUDA.
- Copy requirements first and pip install (cache layer); copy code after (fast rebuilds).
- Copy the artifact (or mount it) from the registry — never build weights at image build time ideally.
- Runtime: non-root user, set PYTHONUNBUFFERED, expose port, set CMD to the serving entrypoint.
- Healthcheck: calls /health; orchestrator uses it for readiness/liveness.
Layers of a model image.
In production
- Model-specific images per architecture (CPU scoring vs GPU LLM serving).
- Multi-stage GPU images: build tooling stage dropped from the final image.
- Sidecar-less memory: artifact mounts keep images small and swap fast.
Best practices
- Pin base images by digest so 'pull latest' can't drift.
- Order layers by change frequency (requirements → code) for cache reuse.
- Run non-root; the model service needs no root privileges.
- Healthcheck matching the orchestrator's probe; keep a distinct readiness for warmup.
Common mistakes
- Baking weights into the image (images become GB-scale, rebuilds slow, secrets leak).
- CUDA base with the wrong cuDNN for the framework — silent crash at showtime.
- Root-running containers and world-writable artifacts.
- No healthcheck, so orchestrators restart healthy-but-stuck replicas.
Common issues
Image works locally, fails in clusterDifferent host arch/base tag, mounts, or env
Pin digest-based images, add a startup self-test, and mirror the cluster's mounts.
GPU container says 'no driver'CUDA runtime vs host driver mismatch
Match cuda base to the cluster driver (nvidia-smi → driver version) and test-drive the image.
Slow startup (minutes)Artifact baked in or no warm cache
Mount artifacts from object storage and cache framework/feature caches across restarts.
Debugging tips
- Reproduce a pod locally: run the exact image with the same mounts and probe /predict.
- Check image layers for accidentally-included training data/secrets (scan before registry push).
- If GPU image OOMs at load, compare CUDA versions in the base vs the framework build.
- Compare image digests across pods — a drifting tag that pinning would have prevented.
Interview checkpoint
Hands-on lab
- Author a two-stage Dockerfile for a model API
- Add a healthcheck and non-root user
- Validate image runs with a probe
Build a Lean Model Image
35 min · intermediate
Quick quiz
GPU inference images differ from CPU images mainly by:
Pinning base-image digests protects against silent registry drift.