Kubernetes for ML
Beginner 13 min read Kubernetes for ML

Kubernetes for ML

Why ML runs on a cluster platform

Kubernetes is the substrate for production ML: it schedules training on GPUs, runs serving deployments, and gives pipelines retries, autoscaling and governance for free. The ML value is in resources and ops — GPUs, quotas, jobs, autoscaling — not in 'running containers'.

kubernetesk8sgpuorchestrationtraining jobsserving
Prerequisites: Kubernetes Architecture Training Pipelines
Before we start

What you will learn

  • Explain what K8s gives ML beyond container running
  • Map the compute model: nodes, pods, resources, quotas
  • Frame training and serving as K8s workloads
Introduction

What is this?

Kubernetes for ML uses the cluster platform to run training and serving at scale: GPUs are schedulable resources, jobs get retries and backoffs, deployments autoscale, and everything is declared in YAML under governance. It's the operational substrate ML teams stop rebuilding.

Motivation

Why do you need it?

  • GPU scarcity is a scheduling problem — K8s allocates and queues GPU capacity.
  • Training jobs need retries, backfills and resource limits; the platform provides them.
  • Serving scales with demand; HPA and autoscaling are native.
  • One declarative control plane covers training, serving and their lifecycle.
Think about it

Mental model

Imagine

A port assigning berths, cranes and staff to ships

Think of

K8s scheduling ML workloads

Ships (jobs) request berths (GPUs) and cranes (CPU/mem); the port (scheduler) assigns capacity, queues what can't dock yet (queuing), and re-schedules after delays. One harbor-master, many cargo types.

Mechanics

How it works

  • Nodes declare capacity (GPU counts, CPU, memory); the scheduler matches pods to nodes.
  • Training runs as Jobs or Kubeflow Operators (PyTorchJob/TFJob) with leader/worker topology.
  • Resource requests and limits make utilization predictable; quotas cap teams.
  • Serving runs as Deployments behind Services, autoscaled by HPA.
  • Storage abstractions (PV/PVC) and secrets/configMaps configure workloads declaratively.

K8s hosting the ML compute estate.

Declare workload
Scheduler
Execute
Autoscale/retry
Expose + log
Real world

In production

  • A team schedules nightly training on shared GPU nodes with team quotas.
  • Serving models as KServe InferenceServices autoscaled by traffic.
  • A Kubeflow-style operator runs distributed PyTorch training with retries built in.
Guidelines

Best practices

  • Declare resource requests AND limits for every ML pod.
  • Use K8s jobs/operators for training, not cron-attached to dev pods.
  • Namespace and quota per team so GPU noise can't sink the cluster.
  • Put storage as persistent volumes; never depend on pod-local disks for artifacts.
Watch out

Common mistakes

  • No resource requests → the scheduler treats every GPU pod as equal.
  • Training on dev pods with no job semantics — no retry, no history.
  • Per-team GPU allocations ungoverned, so one experiment starves the rest.
  • Artifacts left on pod-local storage and lost on reschedule.
When things break

Debugging tips

  • Pod Pending — check requests vs node allocatable and any quota remaining.
  • GPU mismatch — driver/toolkit images vs runner; verify in the node's allocated devices.
  • Job silently failing — look at Job backoffLimit and pod retry logs, not the pod name.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Kubernetes' core offering to ML is…

2

Every ML pod should declare resource requests and limits.

Go deeper

Official documentation

Kubernetes documentation
docs
Kubeflow docs
docs