K8s for Training

• Run training as Jobs with backoff and restart policy
• Distinguish single-pod vs distributed training objects
• Handle checkpoints and cleanup for training pods

• Jobs give retries/backoff to flaky training that plain pods never had.
• Distributed topologies (worker/ps) are encoded in the workload, not scripts.
• Cluster-managed storage and cleanup keep training reproducible and tidy.
• Operators handle the hard parts — failure, eviction, gang scheduling.

Individual workers (pods) do shifts; the foreman (operator) assigns roles (leader/worker), coordinates concrete pours (gradient sync) and replaces any worker who doesn't show. Burned-out crew get rebuilt overnight, and the schedule (checkpoint) continues.

• A Job YAML declares the image, replicas, resources and restart/backoff policy.
• For distributed runs, an operator (e.g. Kubeflow) creates leader + worker pods with a spec.
• The scheduler co-schedules partition-workers for gang-style parallelism.
• Checkpoints stream to persistent storage; failures resk from the checkpoint.
• Completed runs clean up pods while preserving logs and artifacts.

• A nightly fine-tune runs as a Job with backoffLimit=3 and auto-captured logs.
• A PyTorchJob spreads 8 GPUs across workers with gang scheduling via a queue.
• Checkpoints on shared storage let an evicted worker resk instead of restart.

• Declare backoffLimit and activeDeadline so runaway jobs can't burn forever.
• Choose operator-per-framework (PyTorchJob, TFJob) over hand-rolled distributed scripts.
• Stream checkpoints to persistent storage; use the pod only for compute.
• Set cleanup TTLs / finalizers so finished training doesn't litter the namespace.

• Running distributed training as N independent pods with no topology — 'the loss didn't sync'.
• No restart policy — an evicted worker silently halves the world.
• Checkpoints on local disk — an eviction or reschedule loses the training run.
• No deadline: a hang trains forever and bills the whole namespace.

• Workers lag or desync — check the shared-rank/backend config, not the model code.
• Job stuck in backoff loop — read the failed pod's exit reason and logs; often a config or image issue.
• Gang job won't start — too few schedulable GPUs; verify quota and node availability.