highAdvancedRED#alb#504#timeout#latency

ALB returns 504 Gateway Timeout — backend too slow

Requests return 504 from the load balancer at steady rate

Likely source:performanceconnectivity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Requests return 504 from the load balancer at steady rate
  • Specific slow endpoints time out while others work
  • Backend is up but takes longer than the LB idle timeout

The 504 is the LB telling you it waited its full budget. Raising idle_timeout treats the symptom; a backlog of genuinely slow endpoints will surface latency elsewhere — fix the slow path too.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • LB attributes (idle timeout)

    aws elbv2 describe-load-balancer-attributes --load-balancer-arn <LB-ARN>

  • Backend response time

    aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=LoadBalancer,Value=<LB-ARN> Name=TargetGroup,Value=<TG-ARN> --start-time <ISO> --end-time <ISO> --period 300 --statistics Maximum

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Only long requests (reports, uploads, streams) 504
  • Idle timeout configured at or below app p99

Confirm before fixing

aws elbv2 describe-load-balancer-attributes --load-balancer-arn <LB-ARN>
READ

Current idle timeout

Fix

aws elbv2 modify-load-balancer-attributes --load-balancer-arn <LB-ARN> --attributes Key=idle_timeout.timeout_seconds,Value=<LARGER-SECONDS>
CHANGE

Raise the idle timeout, or split the workload so streams use a dedicated LB

Expected: Attributes updated

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws elbv2 describe-load-balancer-attributes --load-balancer-arn <LB-ARN>
READ

Read the timeout baseline

2
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=TargetGroup,Value=<TG-ARN> --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 300 --statistics Maximum
READ

Quantify backend latency

3
aws elbv2 modify-load-balancer-attributes --load-balancer-arn <LB-ARN> --attributes Key=idle_timeout.timeout_seconds,Value=<LARGER-SECONDS>
CHANGE

Raise the timeout to match real workloads

4
aws ssm send-command --instance-ids <TARGET-INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["time curl -s -o /dev/null http://localhost:<PORT>/<ENDPOINT>"]'
READ

Keep the app-latency investigation going from the host

Prevention

  • Set idle_timeout above your worst real endpoint, and alarm on TargetResponseTime.
  • Move long-running work to async jobs with a 202 + poll pattern.
  • Profile slow endpoints (db queries, cold caches) before they cross the timeout line.

Related runbooks

Same stack, same failure family — worth a skim.

ALB returns 503 — no healthy targetscritical

HTTP 503 Service Unavailable from the load balancer

AWSIntermediateRED3 causes
Open runbook
RDS max_connections reached — apps rejectedhigh

Apps error with 'too many connections' from the DB client

AWSIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.