criticalIntermediateRED#http#502#proxy#upstream#nginx

HTTP 502 Bad Gateway -- proxy cannot reach or connect to the upstream service

curl returns HTTP 502 Bad Gateway

Likely source:connectivityavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • curl returns HTTP 502 Bad Gateway
  • nginx error log shows 'upstream connection refused' or 'no live upstreams'
  • Browser shows 502 error page from the reverse proxy

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Proxy error log

    tail -100 /var/log/nginx/error.log | grep -i 'upstream|502'

  • Upstream service status

    curl -v http://<UPSTREAM_HOST>:<UPSTREAM_PORT>/healthz

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • curl to upstream directly returns connection refused
  • ss -tlnp on the upstream host shows no listener on the expected port

Confirm before fixing

curl -v http://<UPSTREAM_HOST>:<UPSTREAM_PORT>/healthz 2>&1 | head -10
READ

Test direct upstream connectivity

Fix

systemctl start <upstream-service>
CHANGE

Start the upstream service

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
tail -100 /var/log/nginx/error.log | grep -i 'upstream|502'
READ

Read the proxy error log

2
curl -v http://<UPSTREAM_HOST>:<UPSTREAM_PORT>/healthz
READ

Test the upstream directly

3
nc -zv <UPSTREAM_HOST> <UPSTREAM_PORT> -w5
READ

Test TCP connectivity to the upstream

4
systemctl start <upstream-service>
CHANGE

Restart the upstream service if it is down

Prevention

  • Set up active health checks on the proxy to quickly detect and remove unhealthy backends.
  • Monitor upstream availability independently from the proxy.
  • Use connection pooling and retry logic in the proxy configuration.
  • Alert on 502 rate > 1% of total requests.

Related runbooks

Same stack, same failure family — worth a skim.

HTTP 504 Gateway Timeout -- upstream service too slow, proxy timeout exceededhigh

curl returns HTTP 504 Gateway Timeout after exactly 60s (default nginx proxy timeout)

NetworkingIntermediateRED3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.