Training Infrastructure

• Design a training platform: containers, queues, scheduling and GPUs
• Explain determinism, resource isolation and cost control on shared compute
• Recognize when distributed training is worth it

• Models and data grow faster than laptops; training needs determinism that notebooks can't give.
• Shared GPUs without scheduling cause collisions, blasts of 'my job OOMed' and untracked spend.
• Containerized environments end 'works on my machine' version rot across runs.
• Reproducibility in an audit/production context demands pinned environments and logged runs.

Everyone cooking simultaneously burns dinner and fights for burners. A restaurant with reservations (scheduling), standardized recipes (containers) and a prep order (orchestration) gets more meals cooked in less time — and you can say exactly what each dish contained.

• Containerize: image pins Python, CUDA and library versions; the image IS the training environment.
• Schedule: jobs request CPUs/GPUs and memory; a scheduler (Kubernetes jobs, Slurm) queues, retries and evicts.
• Run: orchestrators mount datasets, set seeds and env vars, write metrics and artifacts to a tracking server.
• Scale: one GPU per job by default; distributed strategy (multi-worker) only when a single GPU is insufficient.
• Govern: budget per team/experiment, quotas, spot-instance policies and usage dashboards.

• A research team runs hundreds of hyperparameter trials on a Kubernetes GPU pool, each job a container with pinned image and a 2-GPU request.
• An e-commerce ML team uses spot instances for preemptible batch training, keeping interactive GPU quotas for critical runs.
• A large-scale recommendation model trains distributed across 8 nodes with gradient synchronization when one GPU is too slow.

• Pin the full env: Python, CUDA, framework versions, plus a lockfile — and record it per run.
• Give every run a unique ID and log code hash, dataset version and hyperparameters as structured metadata.
• Set resource requests/limits so a runaway job can't starve neighbors.
• Keep datasets versioned and mounted read-only; never train on mutable files.
• Track utilization and cost per team before you scale the platform.

• Running GPU jobs on the CPU pool 'because it's free' — orders of magnitude slower and still costs.
• Installing packages at job start via pip inside containers, reintroducing version drift.
• Ignoring OOM kills: a transient spike kills a 10-hour run without checkpointing.
• Pretending multi-GPU/distributed is free — communication overhead can make it slower.

• Reproduce locally or on a tiny sample first; GPU queue time is too expensive a debugger.
• Check GPU utilization (nvidia-smi) — low utilization means I/O or serialization bottleneck, not more GPUs.
• Enable checkpointing with resume; most long-job failures are transient (eviction, OOM, node loss).
• Compare two identical run IDs with different images to isolate dependency-induced nondeterminism.