highProductionSymptom-first#eks#nodegroup#nodes#eksctl

EKS nodegroup instances not joining the cluster

New nodegroup instances launch but never appear in kubectl get nodes

Likely source:connectivityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • New nodegroup instances launch but never appear in kubectl get nodes
  • Nodegroup status CREATE_FAILED or Degraded
  • CloudFormation stack for the nodegroup ends in ROLLBACK

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Nodegroup status

    aws eks describe-nodegroup --cluster-name <CLUSTER-NAME> --nodegroup-name <NODEGROUP-NAME> --query 'nodegroup.{Status:status,Health:health,Role:nodeRole}'

  • Cluster endpoint

    aws eks describe-cluster --name <CLUSTER-NAME> --query 'cluster.{Endpoint:endpoint,Version:version,Status:status}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • describe-nodegroup shows issue ClusterUnreachable
  • role lack AmazonEKSWorkerNodePolicy

Confirm before fixing

aws iam list-attached-role-policies --role-name <NODE-ROLE-NAME> --query 'AttachedPolicies[].PolicyName'
READ

Attached policies on the node role

Fix

aws iam attach-role-policy --role-name <NODE-ROLE-NAME> --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
CHANGE

Attach the worker node policy (repeat for AmazonEKS_CNI_Policy and AmazonEC2ContainerRegistryReadOnly)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws eks describe-nodegroup --cluster-name <CLUSTER-NAME> --nodegroup-name <NODEGROUP-NAME> --query 'nodegroup.{Status:status,Issues:health.issues,Role:nodeRole}'
READ

Read the issue code

2
aws iam list-attached-role-policies --role-name <NODE-ROLE-NAME>
READ

Audit the node role

3
aws iam attach-role-policy --role-name <NODE-ROLE-NAME> --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
CHANGE

Fix policy gaps

4
aws eks update-nodegroup-config --cluster-name <CLUSTER-NAME> --nodegroup-name <NODEGROUP-NAME>
CHANGE

Roll the nodegroup after fixes (or delete/recreate the nodegroup)

Prevention

  • Create node roles via IaC with exactly the three EKS managed policies.
  • Keep node subnets in the same VPC with NAT/VPC-endpoint egress in place.
  • Wire cluster+node SG pairs explicitly and review them in the security review.

Related runbooks

Same stack, same failure family — worth a skim.

EKS nodes/roles can't authenticate — aws-auth missing entrycritical

kubectl returns 'could not get token: AccessDeniedException' for a role

AWSAdvanced3 causes
Open runbook
Private subnet without NAT — no outbound internethigh

Instances in private subnets cannot reach package repos or APIs

AWSFoundation3 causes
Open runbook
New ASG instances failing to launch — quotas and AMIcritical

ASG scale-out spins up instances that immediately terminate

AWSProduction3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.