highAdvancedUSE#compute engine#persistent disk#detaching#storage

GCE persistent disk stuck in DETACHING state

gcloud compute instances describe shows a disk in state DETACHING

Likely source:availabilityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • gcloud compute instances describe shows a disk in state DETACHING
  • A detach operation hangs and reports running for a long time
  • gcloud compute instances attach-disk fails with 'disk already in use'
  • MIG instance replacement stalls waiting on a stale disk attachment

DETACHING almost always means the guest still has the filesystem mounted or busy. Unmounting inside the VM before issuing the detach from the cloud clears the vast majority of these cases.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Disk attachment state

    gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(disks)'

  • Detach operation status and error

    gcloud compute operations list --filter='operationType=compute.disks.detach' --format='table(name,status,error,insertTime)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • The instance is RUNNING and healthy but the detach hangs
  • The VM host is doing heavy writes to the disk (from monitoring)
  • Detach started without an orderly unmount in the guest

Confirm before fixing

gcloud compute ssh <INSTANCE> --zone=<ZONE> --command='sudo lsof +D /mnt/<MOUNT> 2>/dev/null || sudo fuser -vm /mnt/<MOUNT>'
READ

Check for open files/handles on the mount from inside the guest

Expected: A list of processes still using the mount

Fix

gcloud compute ssh <INSTANCE> --zone=<ZONE> --command='sudo umount /mnt/<MOUNT> && sudo sync'
CHANGE

Unmount and flush the filesystem before detaching

Expected: umount and sync complete without errors

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='yaml(disks)'
READ

Confirm which disk is stuck DETACHING

2
gcloud compute ssh <INSTANCE> --zone=<ZONE> --command='sudo umount /mnt/<MOUNT> && sudo sync'
CHANGE

Unmount + flush the filesystem in the guest

3
gcloud compute instances detach-disk <INSTANCE> --zone=<ZONE> --disk=<DISK> --device-name=<DEVICE>
CHANGE

Retry the detach after the guest is clean

4
gcloud compute instances describe <INSTANCE> --zone=<ZONE> --format='value(disks[?state==DETACHING])'
READ

Verify no disk is left in DETACHING

Prevention

  • Unmount disks cleanly in the guest before detaching or resizing groups.
  • Use lifecycle hooks / pre-stop scripts to flush and unmount on scale-in.
  • Track disk state with alerts on DETACHING/ATTACHING lasting beyond a few minutes.
  • Do not snapshot and detach the same disk concurrently.

Related runbooks

Same stack, same failure family — worth a skim.

GCE instance stuck in PROVISIONING statehigh

gcloud compute instances list shows PROVISIONING for minutes

Google CloudIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.