highIntermediateRED#reset#rst#backend#socket#overload

Connection reset by peer -- RST received from server, backend overload or bug

curl: (56) Connection reset by peer

Likely source:connectivityavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • curl: (56) Connection reset by peer
  • Application logs show ECONNRESET errors intermittently under load
  • tcpdump shows a RST packet from the server after the connection is established

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • RST packets in a capture

    tcpdump -i any host <SERVER> and port <PORT> -c 20 'tcp[tcpflags] & (tcp-rst) != 0' -nn

  • Server-side socket stats

    ss -s && ss -tnp | grep <PORT>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • ss shows a high SYN_RECV or ESTABLISHED count
  • RSTs spike during traffic bursts
  • somaxconn is at the default 128

Confirm before fixing

cat /proc/sys/net/core/somaxconn && ss -tnp | grep LISTEN | awk '{print $4}' | sort | uniq -c | sort -rn | head
READ

Check the backlog limit and per-port listen queue depths

Fix

sysctl -w net.core.somaxconn=4096
CHANGE

Raise the listen backlog limit

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
tcpdump -i any host <SERVER> and port <PORT> -c 10 -nn 'tcp[tcpflags] & (tcp-rst) != 0'
READ

Capture RST packets to confirm server-originated resets

2
ss -s && ss -tnp | grep <PORT>
READ

Check socket state and listen backlog

3
cat /proc/sys/net/core/somaxconn
READ

Read the backlog limit

4
sysctl -w net.core.somaxconn=4096
CHANGE

Raise the backlog limit if it is the bottleneck

Prevention

  • Increase somaxconn and application listen backlog for high-traffic services.
  • Monitor TCP RST counters (netstat -s | grep reset) and alert on spikes.
  • Use connection pooling on the client side to reduce churn.
  • Implement graceful shutdown to drain connections instead of resetting them.

Related runbooks

Same stack, same failure family — worth a skim.

TCP connection refused -- nothing listening on the port or explicit REJECT rulehigh

curl: Connection refused immediately (no timeout delay)

NetworkingFoundation3 causes
Open runbook
HTTP 502 Bad Gateway -- proxy cannot reach or connect to the upstream servicecritical

curl returns HTTP 502 Bad Gateway

NetworkingIntermediateRED3 causes
Open runbook
conntrack table full -- nf_conntrack: table full, dropping packetcritical

dmesg shows 'nf_conntrack: table full, dropping packet'

NetworkingAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.