highIntermediateRED#http#504#timeout#upstream#proxy

HTTP 504 Gateway Timeout -- upstream service too slow, proxy timeout exceeded

curl returns HTTP 504 Gateway Timeout after exactly 60s (default nginx proxy timeout)

Likely source:performanceavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • curl returns HTTP 504 Gateway Timeout after exactly 60s (default nginx proxy timeout)
  • nginx error log shows 'upstream timed out (110: Connection timed out) while reading response header'
  • Clients experience intermittent timeouts only for specific slow endpoints

504 specifically means the proxy connected to the upstream but didn't get a response in time. If the proxy couldn't connect at all, you'd see 502 instead.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Proxy timeout configuration

    grep -r 'proxy_read_timeout|proxy_connect_timeout' /etc/nginx/conf.d/

  • Upstream response time

    curl -w 'time_total: %{time_total}\n' -o /dev/null -s http://<UPSTREAM>/slow-endpoint

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Upstream response time > 60s
  • Upstream logs show slow queries or API calls

Confirm before fixing

curl -w 'time_total: %{time_total}\n' -o /dev/null -s http://<UPSTREAM_HOST>:<UPSTREAM_PORT>/<endpoint>
READ

Measure the actual upstream response time

Fix

echo 'Optimize the slow query or external API call in the upstream application.'
READ

Document the upstream performance fix

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -w 'time_total: %{time_total}\n' -o /dev/null -s http://<UPSTREAM_HOST>:<UPSTREAM_PORT>/<endpoint>
READ

Measure upstream response time

2
grep -E 'proxy_read_timeout|proxy_connect_timeout' /etc/nginx/conf.d/*.conf
READ

Check proxy timeout settings

3
sed -i 's/proxy_read_timeout 60s/proxy_read_timeout 120s/' /etc/nginx/conf.d/<file>.conf && nginx -t && systemctl reload nginx
CHANGE

Increase the timeout and reload if the upstream needs more time

Prevention

  • Set proxy timeouts based on the worst-case upstream response time (P99 + buffer).
  • Implement request timeouts in the upstream application to fail fast.
  • Use circuit breakers in the proxy to stop sending requests to a slow upstream.
  • Alert on upstream response time P99 approaching the proxy timeout.

Related runbooks

Same stack, same failure family — worth a skim.

HTTP 502 Bad Gateway -- proxy cannot reach or connect to the upstream servicecritical

curl returns HTTP 502 Bad Gateway

NetworkingIntermediateRED3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.