highAdvancedSymptom-first#nat#idle-timeout#keepalive#conntrack#websocket

Idle connections silently dropped -- NAT/firewall timeout killing long-lived connections

WebSocket or long-poll connection drops after ~300s or ~3600s of inactivity

Likely source:connectivityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • WebSocket or long-poll connection drops after ~300s or ~3600s of inactivity
  • SSH sessions to remote hosts hang after being idle for several minutes
  • Database connections from an app to a remote DB fail periodically with no server-side error

The most common idle timeouts are 300s (AWS NLB), 600s (AWS classic ELB), and 3600s (GCP). Always configure keepalive probes below the lowest timeout in the path.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Conntrack state for the connection

    conntrack -L | grep <PORT>

  • NAT timeout settings

    sysctl net.netfilter.nf_conntrack_tcp_timeout_established

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Connection dies after a predictable idle period (e.g. 300s for cloud NAT)
  • conntrack entry disappears from the table during the idle window

Confirm before fixing

conntrack -E -p tcp --dport <PORT> 2>/dev/null | head -30
READ

Watch conntrack events in real time to see the entry expire

Fix

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400
CHANGE

Increase the established timeout to a larger value

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
conntrack -L -p tcp --dport <PORT> 2>/dev/null | head -10
READ

Check if the connection is in the conntrack table

2
sysctl net.netfilter.nf_conntrack_tcp_timeout_established
READ

Read the NAT timeout

3
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400
CHANGE

Increase the timeout if needed

4
tcpdump -i any host <REMOTE_HOST> and port <PORT> -c 10 -nn
READ

Capture to verify keepalive behavior

Prevention

  • Enable TCP keepalive probes on all long-lived connections (DB, WebSocket, SSH).
  • Set keepalive interval below the lowest NAT/LB idle timeout in the path.
  • Increase conntrack established timeout if running a NAT gateway on Linux.
  • Monitor conntrack table utilization to prevent drops under load.

Related runbooks

Same stack, same failure family — worth a skim.

conntrack table full -- nf_conntrack: table full, dropping packetcritical

dmesg shows 'nf_conntrack: table full, dropping packet'

NetworkingAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.