highIntermediateSymptom-first#images#imagepullpolicy#registry#rollout

imagePullPolicy Always + immutable tag — pull fails after registry change

New pods fail to pull image after a rollout, even though the tag exists

Likely source:availabilitydeployment

Focused incident mode — strips away everything but this runbook.

Symptoms

  • New pods fail to pull image after a rollout, even though the tag exists
  • ImagePullBackOff on a tag that was previously working
  • Registry returns 404 or manifest unknown for the immutable tag

imagePullPolicy Always is correct for 'latest', but dangerous for immutable tags if the registry is unreliable. Match policy to your tagging strategy.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • imagePullPolicy setting

    kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].imagePullPolicy}'

  • Image reference

    kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • docker manifest inspect returns not found
  • Tag was pushed once but registry has retention policies

Confirm before fixing

docker manifest inspect <registry>/<image>:<tag>
READ

Confirm the tag is gone

Fix

docker tag <image>:<tag> <registry>/<image>:<tag> && docker push <registry>/<image>:<tag>
CHANGE

Re-push the image to the registry

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl describe pod <pod> -n <namespace> | grep -A5 Events
READ

Read the pull error

2
docker manifest inspect <registry>/<image>:<tag>
READ

Prove the tag exists in the registry

3
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].imagePullPolicy}'
READ

Confirm pull policy

4
kubectl patch deployment/<deploy> -n <namespace> --type merge -p '{"spec":{"template":{"spec":{"containers":[{"name":"<c>","imagePullPolicy":"IfNotPresent"}]}}}}'
CHANGE

Switch to IfNotPresent if the tag is immutable

Prevention

  • Use immutable tags (commit SHAs) with imagePullPolicy IfNotPresent.
  • Never configure registry GC to remove tags referenced by running deployments.
  • Set up image mirror/cache (e.g., Harbor proxy cache) to survive registry outages.
  • Use imageDigestReferences instead of tags where possible.

Related runbooks

Same stack, same failure family — worth a skim.

ImagePullBackOff / ErrImagePull — pod can't fetch its imagehigh

Pod stuck in ImagePullBackOff or ErrImagePull

KubernetesFoundation3 causes
Open runbook
Deployment rollout stuck — progress deadline exceededhigh

kubectl rollout status hangs on 'Waiting for rollout to finish'

KubernetesIntermediate2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.