Compute
Intermediate 12 min read Compute

Spot and Preemptible Instances

Cheap compute that can vanish

Spot (AWS), and preemptible (GCP) compute is spare capacity sold at a steep discount — but the provider may reclaim it. Idempotent, resumable workloads harvest real savings; fragile ones get evicted.

spotpreemptibleinterruptioncapacitydiscountcheckpoint
Prerequisites: Compute Families and Sizing
Before we start

What you will learn

  • Explain how spot pricing and reclamation work
  • Design workloads that tolerate interruption
  • Compare spot, capacity blocks and preemptible semantics
In 60 seconds

Providers resell idle capacity at a discount (often 60-90%). You get it until the provider needs it back — which can be minutes-to-hours away. State must be external and work resumable.

Introduction

What is this?

AWS EC2 Spot reaches ~1-5 min warning (or capacity rebalance for some types) and discounts of up to 90%. Azure Spot VMs offer scale sets with eviction policies (delete/deallocate). GCP preemptible/spot instances run up to 24h (preemptible) or until preempted (spot) with a 30s notice. Interruption reclaims a random slice before terminating; capacity pools by type/region/pricing history. Reserved spot capacity (Capacity Blocks) buys guaranteed windows for jobs like ML.

Motivation

Why do you need it?

  • Batch, stateless, and resumable work pays a fraction of On-Demand.
  • CI fleets and large renders finish fast and cheap when spot is abundant.
  • Cost optimization reports call out idle On-Demand that spot replaces.
  • The burden: interruption handling must be engineered, not hoped.
Think about it

Mental model

Imagine

Standby airplane seats at half price, gate changes with notice

Think of

Spot instances

You fly cheaply when there's spare capacity; if the flight fills (capacity is recalled), you're rebooked (reclaimed) — pack a travel bag that fits rescheduling (external state).

Mechanics

How it works

  • Spot Fleet/request: pick types and a price cap; the fleet launches from available capacity.
  • Pricing is set by supply/demand; you get current market price or the max you cap.
  • Reclamation: the provider sends a 2-minute warning (AWS metadataparece) then reclaims confused discounts.
  • Diversification across families/regions/AZs raises your access rate.
  • Workloads checkpoint to durable storage; workers rejoin queues after preemption.

Spot lifecycle.

Request spot
Launch
Run at discount
Interruption notice
Rejoin capacity
Hands-on

Examples

Interruption-tolerant worker

Interruption-tolerant worker
# advertise, do work in chunks, checkpoint on termination hook (AWS)
python <<'EOF'
import socket, os
# worker: claim a chunk from the queue
chunk = claim_next_chunk()
# process in small units, checkpoint after each
for i, record in enumerate(chunk):
    process(record)
    checkpoint(record_id=i)
    if os.path.exists("/tmp/terminate"):   # set by lifecycle hook
        break
EOF

Chunked, checkpointed, queue-based work re-queues safely after eviction — spot-safe by construction.

Real world

In production

  • A CI fleet on spot: builds are idempotent, reruns pick up where the environment resets.
  • Genomics pipelines checkpoint per-sample; preempted workers reclaim another draft.
  • Render farms spread across spot capacity blocks across 3 regions with re-queuing.
Guidelines

Best practices

  • Put all durable state in S3/DB/GCS so instances are cattle.
  • Use spot diversification (multiple types + availability zones).
  • Handle interruption grace: drain, checkpoint, and dependency-queue release.
  • Use capacity blocks/reservations for time-critical (ML training) instead of free-floating spot.
Watch out

Common mistakes

  • Running a database or a single web instance on spot.
  • No reclamation handling — jobs silently die mid-write.
  • One instance type in one AZ: worst access rate.
  • Confusing 'fixed price' savings with spot volatility in a cost model.
Root cause analysis

Common issues

Error
Spot fleet can't launch
Root cause

Capacity tight in that type+AZ or price cap too low

Fix

Add more instance types and AZs; raise max price cap; retry with diversity.

Error
Workers die mid-job repeatedly
Root cause

No checkpointing or external state

Fix

Refactor to chunked idempotent units and durable checkpoints before relying on spot.

When things break

Debugging tips

  • Check spot interruption counts in EC2 CloudTrail/Console; spikes signal thin capacity.
  • Rightsize: reclaimed types cluster; diversify and watch 'access rate' metrics.
  • If jobs vanish, verify the external state commit happened before the 2-min cutoff.
  • Log reclamation events to correlate job failures with capacity thinning.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build an Interruption-Tolerant Spot Worker
35 min · advanced
In progress
Objectives
  • Model a spot worker that checkpoints and re-queues
  • Simulate an eviction and verify the resume path
  • Write the AWS interrupTion-handling hook wiring

Build an Interruption-Tolerant Spot Worker

35 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/1
1

What generally causes spot instance reclamation?

Go deeper

Official documentation

AWS — Spot Instances
docs
Azure — Spot VMs
docs
GCP — Preemptible and Spot
docs