Storage and State
The state that must outlive the pods
ML state lives outside pods: model artifacts, checkpoints, feature snapshots and outputs belong on persistent volume and object stores. K8s storage abstractions (PV/PVC) give pods durable access, but the real lesson is never trusting pod-local disks for anything that must survive.
What you will learn
- Use PV/PVC for durable, pod-independent ML state
- Choose object storage for artifacts and checkpoints
- Design for eviction without state loss
What is this?
Storage and state for ML on K8s means the durable home of artifacts, checkpoints, datasets and outputs: persistent volumes (PV/PVC) mounted by pods plus object/blob storage for large artifacts. Pod-local disks are compute scratch, never the source of truth.
Why do you need it?
- Pods are ephemeral; any state they 'own' dies or scatters on reschedule.
- Checkpoints on shared storage are what make eviction recoverable.
- Large model/data artifacts deserve object storage's scale and durability.
- Explicit storage topology (mount vs blob) keeps caching and cost legible.
Mental model
A professor's gradebook kept centrally, not in lecture-hall chairs
ML state on PV/object storage
Filling in a chair's armrest (pod-local disk) is convenient but the class moves rooms and hallways get wiped. The gradebook (PV) and library copies (object store) live elsewhere, so a room change costs nothing.
How it works
- Administrators define PersistentVolumes; pods claim them via PVC with access modes.
- Object storage (S3/GCS) holds large artifacts; pods read/write via client libraries.
- Checkpoints stream periodically to durable storage during training.
- A job's outputs and logs are pushed to durable storage before the pod terminates.
- Stateful components (feature stores, vector DBs) run as StatefulSets on those volumes.
Durable ML state layers.
In production
- Training streams checkpoints to S3 so worker evictions resk, not restart.
- A StatefulSet hosts the feature store with PV-backed persistence and stable identity.
- Batch outputs are uploaded to object storage with run-id prefixes before completion.
Best practices
- Keep pod-local disk for scratch; home all durable state in PV/object stores.
- Stream checkpoints frequently; a 3h checkpoint interval defeats recovery.
- Namespace objects as <run_id>/<artifact> in object storage for lineage.
- Pick access modes that match readers (RWX for shared training mounts where needed).
Common mistakes
- Checkpoints on the pod's local disk, then an eviction erases the run.
- One giant PVC mounted everywhere, bottlenecking and billing weirdly.
- No cleanup/retention on object stores — runs accumulate forever at cost.
- Stateful components shoehorned into stateless Deployments, losing identity.
Debugging tips
- Checkpoint 'lost' — read the mount type; local disk vs PV vs object store explains it.
- Slow training writes — profile storage throughput; object-store PUTs beat PV rewrites for ckpt.
- Pod can't mount — verify PVC exists, is bound, and access mode matches the workload.
Interview checkpoint
Quick quiz
A training run's checkpoints should live…
Object storage is a fine home for large model artifacts with retention policies.