Autoscaling and Spot Instances

• Right-size serving capacity with autoscaling policies
• Use spot/preemptible capacity for interrupt-tolerant work
• Engineer checkpoint-and-resume against evictions

• Serving capacity sized to peak is 60-80% idling at trough.
• Spot can cut training compute 50-70% when interruptions are survivable.
• Autoscaling yields faster response to new demand without perpetual over-provision.
• Checkpoint-plus-resume converts a scary eviction into a mild delay.

Keeping five dishwashers on payroll to cover a two-hour rush is idling money all week. Hire for ordinary load, call in burst help (scale-out) for Saturday, and let the shift manager stop the clock when traffic dies. Same is true for GPU capacity and a queue of jobs.

• Serving: autoscale on queue depth/latency with hysteresis — CPAs, min/max replicas.
• Training: scale queue worker pools; burst jobs onto spot during cheap windows.
• Spot: bid/preemptible pricing model, instance profile audited for tolerance.
• Checkpoint-and-resume: periodic snapshots (every N steps/minutes); resume from latest.
• Mix: reserve the critical floor, spot the burst headroom, on-demand the spot of last resort.

• A serving mesh autoscales the model replicas on queue-depth with 30s hysteresis — peak and trough both cheap.
• A nightly retraining job bursts onto spot GPU with 10-minute checkpoints; a mid-run eviction costs 10 minutes, not a rerun.
• A batch experimentation queue prefers spot; the few jobs that matter hold a small reserved floor.

• Autoscale on a lagging metric (queue depth) not just raw traffic.
• Apply hysteresis so scale-in/out doesn't flap on noise.
• Audit every workload's interrupt tolerance before allowing spot.
• Checkpoint frequency sized so eviction cost stays tiny and bounded.

• Peak-ever-provisioned serving capacity, permanently.
• Spot for an interrupt-hostile workload, then surprise restarts every run.
• No checkpoints on spot jobs — eviction = full rerun, saving nothing.
• Autoscale flap from tight thresholds — buying and billing churn through scale-out/in.

• Scale-out storm then sudden idle — thresholds too tight; add hysteresis and smoothing.
• Warm-up latency at scale-out — degree 0 replicas cold-start; pre-warm or buffer the surge.
• Spot evictions concentrated at peak hours — bid/index mat poor; schedule the tolerant jobs outside density windows.