Kubernetes for ML
Intermediate 15 min read Kubernetes for ML

ML Pods and Deployments

The basic unit of accidental ML infrastructure

ML pods are the unit of compute: containers with model/training code, resource requests, GPU allocation and probes. Deployments add declarative rollouts and self-healing — the pattern serving and small training both ride on.

poddeploymentresource limitsprobesreadinessdeclarative
Prerequisites: Kubernetes for ML Pods
Before we start

What you will learn

  • Write an ML pod with requests, limits and GPU fields
  • Use probes so rollout health is honest
  • Declare rollouts and rollbacks via Deployments
Introduction

What is this?

An ML pod packages one workload — model server or training worker — with resources, volumes and probes; a Deployment adds declarative rollout semantics: desired replica count, rolling updates, self-healing and rollback. This is the bread and butter object for serving and small training.

Motivation

Why do you need it?

  • Explicit resource requests make scheduling predictable and clusters sane.
  • Probes let the cluster know a worker is actually healthy, not just running.
  • Deployments make 'update the model' a declarative action with rollback.
  • Self-healing replaces 'a pod died, who noticed'.
Think about it

Mental model

Imagine

A catering van crew: roster, kitchen, and a checklist

Think of

An ML pod spec

The roster (containers), supplies (resources), kitchen space (volumes) and a health card (probes) are declared before dispatch. The dispatcher (Deployment) keeps the right number of vans rolling and swaps crews without losing a shift.

Mechanics

How it works

  • The pod spec declares the image, resource requests/limits, GPU count and sidecars.
  • Probes (/live, /ready) gate traffic and restarts.
  • A Deployment defines desired replicas and the update strategy (rolling, max surge).
  • On rollout, new pods must pass readiness before the old ones scale down.
  • If new pods can't become ready, the rollout reports failure and history provides rollback.

A Deployment's rollout lifecycle.

Apply spec
Replicate
Readiness
Rolling update
Healthy deployment
Hands-on

Examples

A minimal serving pod

A minimal serving pod
apiVersion: apps/v1
kind: Deployment
metadata: { name: churn-serve }
spec:
  replicas: 3
  selector: { matchLabels: { app: churn } }
  template:
    metadata: { labels: { app: churn } }
    spec:
      containers:
        - name: model
          image: registry.internal/churn:2026-03-14
          resources:
            requests: { cpu: "500m", memory: 1Gi }
            limits:   { cpu: "2", memory: 4Gi }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }

Three replicas with honest resource bounds and a readiness probe; rollouts and rollbacks come from revision history.

Real world

In production

  • A serving deployment rolls a new model version with a max-surge strategy and probes.
  • A small training job runs as a one-shot Job pod with limits so it can't eat the node.
  • Readiness gating stalls bad rollouts before traffic reaches a broken model.
Guidelines

Best practices

  • Set both requests and limits; ML pods that forget limits can starve the cluster.
  • Add readiness probes that measure serving-real (model loaded), not container-up.
  • Use labels/versions in image tags; never tag model images 'latest'.
  • Cap maxSurge and maxUnavailable so model rollouts are deliberate.
Watch out

Common mistakes

  • Readiness probe = 'Docker runs', so broken-model pods take traffic.
  • No resource requests — the scheduler guesses and GPU pods pile onto one node.
  • Mutating pods instead of Deployments, losing rollback to revision history.
  • Images tagged 'latest' — rollback is archaeology.
When things break

Debugging tips

  • Pod CrashLoop — read the previous container's logs and the probe failure; usually model load vs probe mismatch.
  • Rollout never completes — a new replica can't pass readiness; test the probe locally with curl.
  • Being OOMKilled — set memory limits from profiled load, not a guess.
Interviews

Interview checkpoint

Practice

Hands-on lab

Declare a Health-Gated Model Deployment
35 min · intermediate
In progress
Objectives
  • Write a Deployment with resource requests and probes
  • Validate the YAML semantics headlessly
  • Model the rollout/rollback decision flow

Declare a Health-Gated Model Deployment

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A readiness probe should measure…

2

Resource limits are optional for ML pods.

Go deeper

Official documentation

Kubernetes Deployments
docs
Kubernetes pod lifecycle probes
docs