highIntermediateRED#alb#health-check#target#elbv2

ALB target unhealthy — health check configuration drift

One target shows unhealthy while others stay healthy

Likely source:connectivityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • One target shows unhealthy while others stay healthy
  • New app version breaks health but keeps serving traffic otherwise
  • Health check failures cycle over time (healthy→unhealthy→healthy)

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Per-target reason

    aws elbv2 describe-target-health --target-group-arn <TG-ARN> --query 'TargetHealthDescriptions[].{Id:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason,Detail:TargetHealth.Description}'

  • Health check settings

    aws elbv2 describe-target-groups --target-group-arns <TG-ARN> --query 'TargetGroups[0].{Path:HealthCheckPath,Port:HealthCheckPort,Interval:HealthCheckIntervalSeconds,Timeout:HealthCheckTimeoutSeconds,Threshold:HealthyThresholdCount}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Direct curl to the real endpoint works but health fails
  • Health path returns 404/redirect for the current app version

Confirm before fixing

aws elbv2 describe-target-groups --target-group-arns <TG-ARN> --query 'TargetGroups[0].HealthCheckPath'
READ

What path is configured

Fix

aws elbv2 modify-target-group --target-group-arn <TG-ARN> --health-check-path /healthz
CHANGE

Point the probe at the real health endpoint

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws elbv2 describe-target-health --target-group-arn <TG-ARN> --query 'TargetHealthDescriptions[].TargetHealth'
READ

Baseline states + reasons

2
aws elbv2 describe-target-groups --target-group-arns <TG-ARN> --query 'TargetGroups[0].HealthCheck*'
READ

Review the configured probe

3
aws ssm send-command --instance-ids <TARGET-INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["curl -sv http://localhost:<PORT><PATH>"]'
READ

Reproduce locally

4
aws elbv2 modify-target-group --target-group-arn <TG-ARN> --health-check-path /healthz --health-check-timeout-seconds 10
CHANGE

Align and relax the health check

Prevention

  • Expose a /health endpoint that returns quickly and is exempt from auth.
  • Test health config per release in a canary target group.
  • Alert on any single target unhealthy for > 2 intervals.

Related runbooks

Same stack, same failure family — worth a skim.

ALB returns 503 — no healthy targetscritical

HTTP 503 Service Unavailable from the load balancer

AWSIntermediateRED3 causes
Open runbook
ALB returns 504 Gateway Timeout — backend too slowhigh

Requests return 504 from the load balancer at steady rate

AWSAdvancedRED3 causes
Open runbook
NACL vs security group — connection dropped by the wrong layerhigh

Traffic fails in one direction only (e.g. inbound works, replies dropped)

AWSIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.