highIntermediateSymptom-first#daemonset#scheduling#nodeSelector#affinity

DaemonSet pods not scheduled on expected nodes — selector or affinity mismatch

DaemonSet desiredNumberScheduled < number of eligible nodes

Likely source:availabilityscaling

Focused incident mode — strips away everything but this runbook.

Symptoms

  • DaemonSet desiredNumberScheduled < number of eligible nodes
  • Pods not appearing on specific node pools or taint-excluded nodes
  • kubectl describe daemonset shows 0/5 nodes have the required label

DaemonSets are designed to run on every eligible node. If desiredNumberScheduled is less than the node count, the first thing to check is whether the nodeSelector/affinity labels exist on the target nodes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • DaemonSet status

    kubectl describe daemonset <name> -n <namespace>

  • Node labels and taints

    kubectl get nodes --show-labels | head -5

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • DaemonSet spec has nodeSelector but nodes lack the label
  • Events: 0 nodes match the nodeSelector

Confirm before fixing

kubectl get nodes --show-labels | grep -c '<selector-key>'
READ

Count nodes with the required label

Fix

kubectl label node <node> <selector-key>=<selector-value>
CHANGE

Add the missing label to the nodes

Expected: node/<node> labeled

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl describe daemonset <name> -n <namespace> | grep -A5 'Node Status'
READ

Check which nodes are missing pods

2
kubectl get daemonset <name> -n <namespace> -o jsonpath='{.spec.template.spec.nodeSelector}'
READ

Read the nodeSelector

3
kubectl get nodes --show-labels
READ

Audit node labels against the selector

4
kubectl label node <node> <key>=<value>
CHANGE

Add the missing label

Prevention

  • Label nodes consistently with node pools (e.g., role=worker, role=monitoring).
  • Use kubectl get nodes --show-labels regularly to audit labels.
  • Document required node labels as part of the DaemonSet manifest.
  • Add a post-deploy check that desiredNumberScheduled == numberAvailable.

Related runbooks

Same stack, same failure family — worth a skim.

Pods stuck in Pending state — unschedulablehigh

kubectl get pods shows Pending for minutes

KubernetesIntermediate3 causes
Open runbook
Worker node NotReady — node down or kubelet failurecritical

kubectl get nodes shows NotReady for one or more workers

KubernetesAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.