GPU Scheduling and Quotas
Sharing finite silicon fairly
GPUs are scarce, indivisible, node-pinned resources. Scheduling them means declaring how many each pod needs, queuing when none are free, and capping teams with quotas so one experiment can't sink the fleet.
What you will learn
- Declare GPU requests and understand node-pinning
- Implement queues, preemption and fairness
- Limit per-team GPU spend with quotas
What is this?
GPU scheduling allocates scarce GPU devices to pods: each pod asks for a count (nvidia.com/gpu), the scheduler pins to a node with free GPUs, and a queue admits waiting work. Quotas cap per-team consumption so no single wave of experiments starves the cluster.
Why do you need it?
- GPUs are finite and node-bound — a pod can't straddle nodes for a GPU.
- Without queuing, hungry teams fight over whatever's free with chaos.
- Preemption/backfill recover utilization when a big job finishes early.
- Quotas make GPU spend a governed resource, not a scramble.
Mental model
Airport gates and boarding queues
GPU scheduling
Each gate (GPU) is fixed and can't split (indivisible). Flights (jobs) join a queue until a gate frees; airlines (teams) hold slot limits (quotas) so one airline can't monopolize every gate.
How it works
- Device plugins advertise GPU counts per node; pods request nvidia.com/gpu: N.
- The scheduler matches GPU-ready nodes with the requested count and places the pod.
- Admission/queuing controllers (Kueue, Volcano) queue oversubscribed work.
- Quota objects (ResourceQuota, namespaces) cap per-team GPU totals.
- Preemption and gang scheduling coordinate training jobs that need many GPUs at once.
GPU request → queue → allocation.
In production
- A training queue admits 8-GPU jobs only when 8 GPUs fit on one node or gang.
- Per-team quotas cap nightly experiments so serving always has head-room.
- Backfill runs small jobs during a big job's head/tail to keep GPUs busy.
Best practices
- Require explicit GPU requests; default to none to stop silent hoarding.
- Set team quotas (namespaces) and enforce with ResourceQuota.
- Use queuing (Kueue/Volcano) for oversubscription instead of letting jobs fail Pending.
- Choose gang scheduling for distributed training that needs many GPUs simultaneously.
Common mistakes
- No quota → one team's 64-GPU sweep makes everyone else's pods Pending.
- No queue → burst work shows as a wall of Pending with no fairness order.
- GPU pods without limits — a runaway experiment eats the allocated device and neighbors suffer.
- Ignoring drivers/versions — device plugin vs image mismatch yields 'no GPU'.
Debugging tips
- Pod Pending on GPU — check quota balance AND node allocatable plus device plugin health.
- Gang job stuck — the gang can't form; ensure enough contiguous nodes, or relax gang policy.
- GPU flapping — driver kill/restart logs on the node; verify the CUDA image matches the driver.
Interview checkpoint
Hands-on lab
- Model GPU node-pinning and bin packing
- Add a FIFO-with-priority queue
- Enforce a team quota cap
Schedule GPUs with Queues and Quotas
40 min · advanced
Quick quiz
A GPU can be allocated to a pod only if…
Team quotas exist to keep any one GPU consumer from starving everyone else.