highAdvancedSymptom-first#persistentvolume#reclaim#storage#controller

PV reclaim slow or stuck — controller not picking up Released volumes

PV stays in Released state for hours after PVC deletion

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • PV stays in Released state for hours after PVC deletion
  • Underlying cloud volume (EBS, disk) still exists after PV release
  • New PVC cannot bind because the old PV has not been reclaimed

PV reclaim is handled by the CSI controller. If it is down or rate-limited, Released volumes accumulate and cloud costs grow. Monitor Released PV count as an operational metric.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • PV status and policy

    kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}: {.status.phase} policy={.spec.persistentVolumeReclaimPolicy}{"\n"}{end}'

  • CSI controller pod health

    kubectl get pods -n kube-system | grep -E 'csi|provisioner'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • No CSI controller pods in kube-system
  • PV released but underlying disk still exists

Confirm before fixing

kubectl get pods -n kube-system | grep -E 'csi|ebs'
READ

Check CSI controller pods

Fix

kubectl rollout restart deployment/ebs-csi-controller -n kube-system
CHANGE

Restart the CSI controller

Expected: deployment.apps/ebs-csi-controller restarted

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get pv -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy
READ

List PVs and their reclaim status

2
kubectl get pods -n kube-system | grep -E 'csi|provisioner'
READ

Check CSI controller health

3
kubectl rollout restart deployment/ebs-csi-controller -n kube-system
CHANGE

Restart the CSI controller if unhealthy

4
kubectl patch pv <pv> -p '{"spec":{"claimRef":null}}'
CHANGE

Clear claimRef to force the PV to Available (if appropriate)

Prevention

  • Monitor PV phase and alert on Released state > 5 minutes.
  • Ensure CSI controller has PDBs and is monitored for crashes.
  • Use Delete reclaim policy unless the volume holds data that must survive PVC deletion.
  • Test PVC create/delete cycles in staging to validate the full lifecycle.

Related runbooks

Same stack, same failure family — worth a skim.

PersistentVolume stuck in Released state — reclaim not happeninghigh

kubectl get pv shows status Released for extended period

KubernetesAdvanced3 causes
Open runbook
PersistentVolumeClaim stuck in Pending — no provisionerhigh

kubectl get pvc shows Pending

KubernetesAdvanced2 causes
Open runbook
StatefulSet stuck — ordered readiness, PVC pending, or headless service issuehigh

StatefulSet pods stuck in Pending or ContainerCreating sequentially

KubernetesAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.