highFoundationSymptom-first#jobs#backoff#retries#batch

Job backoffLimitExceeded — retries exhausted, job failed permanently

kubectl get job shows Failed status with backoffLimitExceeded

Likely source:availabilitydeployment

Focused incident mode — strips away everything but this runbook.

Symptoms

  • kubectl get job shows Failed status with backoffLimitExceeded
  • Job pod shows Error or CrashLoopBackOff and is not restarting
  • Job completions never reached, active=0, failed>backoffLimit

A Job with backoffLimitExceeded is a dead end — Kubernetes will not restart it. You must fix the root cause and either patch the backoff or create a new Job.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Job status

    kubectl get job <job> -n <namespace> -o jsonpath='{.status}'

  • Job pod logs

    kubectl logs <job-pod> -n <namespace> --previous

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Job pod exit code is 1 or another nonzero value
  • Logs show application error, missing file, or failed assertion

Confirm before fixing

kubectl get pod <job-pod> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
READ

Read the exit code

Fix

kubectl create job <job>-retry --from=cronjob/<cronjob> -n <namespace>
CHANGE

Re-run the job after fixing the application issue

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get job <job> -n <namespace> -o jsonpath='{.status}'
READ

Read the Job status and conditions

2
kubectl logs <job-pod> -n <namespace> --previous
READ

Read the application error

3
kubectl patch job <job> -n <namespace> --type merge -p '{"spec":{"backoffLimit":6}}'
CHANGE

Raise the backoff limit if failures are transient

4
kubectl create job <job>-retry --from=cronjob/<cronjob> -n <namespace>
CHANGE

Manually re-trigger the job

Prevention

  • Set backoffLimit high enough for transient failures (6 is a reasonable default).
  • Implement application-level retries for flaky external dependencies.
  • Alert on Job failures using kube_job_status_failed metric.
  • Use activeDeadlineSeconds only when the job MUST complete within a window.

Related runbooks

Same stack, same failure family — worth a skim.

CronJob not running — suspended, schedule mismatch, or timezone issuehigh

No new Jobs created for a CronJob past its scheduled time

KubernetesIntermediate3 causes
Open runbook
CrashLoopBackOff — container keeps crashing and restartingcritical

kubectl get pods shows CrashLoopBackOff status

KubernetesFoundation4 causes
Open runbook
Container exits immediately with exit code 0 — entrypoint daemonizes or returnshigh

Pod restarts repeatedly with exit code 0 and status Completed

KubernetesIntermediate2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.