criticalIntermediateSymptom-first#compute engine#preemptible#vm#stopped#autoscaling

Compute Engine VM stopped unexpectedly (preemptible / cost control)

gcloud compute instances list shows the instance in TERMINATED state

Likely source:availability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • gcloud compute instances list shows the instance in TERMINATED state
  • Workload disappears from monitoring without a deploy
  • Cloud Logging shows instance stopping events with no initiator
  • App is unreachable and there is no manual stop on record

Preemptible VMs can be reclaimed with zero notice — never run the only copy of a stateful workload on one. Check scheduling flags before assuming an operator caused the stop.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Instance state and machine type

    gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(status,machineType,scheduling)'

  • Instance operation history

    gcloud compute operations list --filter='targetLink~<INSTANCE>' --format='table(name,operationType,status,insertTime)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • instances describe shows scheduling.preemptible true
  • Audit log shows the stop with a Google-managed or empty principal
  • Similar preemptible instances across the group stopped together

Confirm before fixing

gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(scheduling.preemptible,scheduling.instanceTerminationAction)'
READ

Check preemptible and termination action

Expected: preemptible true, instanceTerminationAction DELETE

Fix

gcloud compute instances update <INSTANCE> --zone=<ZONE> --no-preemptible
CHANGE

Convert to a standard (non-preemptible) instance if the workload is not interruptible

Expected: Updated [<INSTANCE>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(status,scheduling)'
READ

Confirm state and scheduling before acting

2
gcloud compute instances start <INSTANCE> --zone=<ZONE>
CHANGE

Start the instance to restore service

Expected: Instance [<INSTANCE>] starting.

3
gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(status)'
READ

Verify it reaches RUNNING and the app restores

Prevention

  • Run critical workloads on standard VMs; reserve preemptible/spot only for truly interruptible batch jobs.
  • Use Managed Instance Groups with a health check so lost VMs are replaced automatically.
  • Set a nonzero autoscaling minimum for services that must never scale to zero.
  • Alert on compute.instances.stop audit log events for critical instance groups.

Related runbooks

Same stack, same failure family — worth a skim.

GCE instance stuck in PROVISIONING statehigh

gcloud compute instances list shows PROVISIONING for minutes

Google CloudIntermediateUSE3 causes
Open runbook
GCE_INTERNAL_ERROR / zone capacity SERVICE_UNAVAILABLE on startcritical

gcloud compute instances start returns error GCE_INTERNAL_ERROR

Google CloudAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.