K8s for Serving
Stable endpoints and honest health behind the cluster
Serving on K8s is Deployments + Services + autoscaling: images pinned, probes honest, traffic routed through stable endpoints. KServe layers the InferenceService on top; bare K8s still serves plenty of models well.
What you will learn
- Compose Deployment + Service + ingress for a model API
- Make probes and routing safe under model load
- Choose bare K8s vs KServe for a workload
What is this?
Serving on Kubernetes assembles the platform's building blocks into a model API: a Deployment of model servers behind a Service, exposed via ingress, with probes gating traffic and autoscaling following load. KServe formalizes this into an InferenceService; K8s primitives do the same for simple cases.
Why do you need it?
- The platform gives you routing, health, scaling and rollback as primitives.
- A Service gives a stable DNS endpoint regardless of pod churn.
- Ingress handles TLS, pathing and load for external calls.
- Choose the right abstraction level: primitives where KServe is overkill.
Mental model
A hotel's front desk, rooms and lobby doors
K8s serving stack
Rooms (model pods) get cleaned and swapped; the front desk (Service) always answers at the same number; the lobby door (ingress) controls who enters. A guest never cares whether room 402 changed since checkout.
How it works
- The Deployment keeps N model-server replicas with probes and resources.
- A Service selects those pods and provides a stable cluster endpoint.
- Ingress exposes the API externally with TLS and routing rules.
- HPA autoscales replicas on metrics; readiness gates traffic to healthy pods.
- Rollouts swap model versions through revision history with rollback.
K8s primitives forming a serving stack.
In production
- A churn API rides a 3-replica Deployment behind a Service with p95 monitored.
- An ingress path routes /v1 vs /v2 model versions during slow rollout.
- HPA keeps the fleet at 3-8 pods while a Service absorbs pod churn.
Best practices
- Point probes at model-ready condition, not container-up.
- Pin image tags; rolling updates + readiness give safe model swaps.
- Set HPA bounds and stabilization; GPU-bound serving scales in whole-GPU steps.
- Route version on the path and version-response contracts for safe rolling.
Common mistakes
- Probes pass before the model is loaded → traffic into the void.
- Serving multiple versions on the same Service path with no routing logic.
- Autoscaler on CPU for a GPU-bound model (signal divorced from bottleneck).
- Versionless 'latest' images; rollback becomes archaeology again.
Debugging tips
- HTTP 503s — readiness failing; check the probe path endpoint directly from a pod.
- Traffic to wrong version — check Service selector labels vs pod labels.
- Latency spikes after rollout — new replica scale-up + model load competing with old traffic.
Interview checkpoint
Quick quiz
The object that provides a stable endpoint over churning pods is…
Readiness probes that pass too early route traffic into broken model pods.