criticalIntermediateUSE#disk#images#layers#prune#storage

Disk full — dangling images and layers consuming /var/lib/docker

docker: error response from daemon: no space left on device

Likely source:data-lossperformancescaling

Focused incident mode — strips away everything but this runbook.

Symptoms

  • docker: error response from daemon: no space left on device
  • df -h /var/lib/docker shows 100% usage
  • docker build fails because the daemon cannot write layers

docker system prune -af --volumes is destructive and removes ALL unused data. Always run docker system df first to see what will be reclaimed.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Disk usage on Docker root

    df -h /var/lib/docker

  • Docker disk breakdown

    docker system df -v

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • docker images -f dangling=true shows many untagged images
  • docker system df shows high Images reclaimable
  • CI/CD builds new images without pruning old ones

Confirm before fixing

docker images -f dangling=true
READ

List all dangling (untagged) images

Fix

docker image prune -f
DESTRUCTIVE

Remove all dangling images

Expected: Deleted: <N> images, reclaimed <SIZE>

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
df -h /var/lib/docker
READ

Confirm disk full

2
docker system df
READ

Break down what is using disk

3
docker image prune -f
DESTRUCTIVE

Remove dangling images

4
docker container prune -f
DESTRUCTIVE

Remove stopped containers

5
docker system prune -af --volumes
DESTRUCTIVE

Nuclear option: remove all unused images, containers, networks, and volumes

6
docker system df
READ

Verify disk is freed

Prevention

  • Set up a cron job: docker system prune -f --filter 'until=24h' to auto-clean daily.
  • Use --squash in docker build to merge layers and reduce image size.
  • Configure Docker daemon log rotation and container log max-size in /etc/docker/daemon.json.
  • Alert on /var/lib/docker usage > 80% via node_exporter or equivalent.

Related runbooks

Same stack, same failure family — worth a skim.

Disk full — filesystem at 100% usagecritical

No space left on device errors in application logs

LinuxFoundationUSE3 causes
Open runbook
ImagePullBackOff / ErrImagePull — pod can't fetch its imagehigh

Pod stuck in ImagePullBackOff or ErrImagePull

KubernetesFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.