highAdvancedSymptom-first#services#kube-proxy#iptables#clusterip

Service ClusterIP unreachable — kube-proxy stale iptables or conntrack

curl to ClusterIP from a pod returns connection refused or timeout

Likely source:connectivityavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • curl to ClusterIP from a pod returns connection refused or timeout
  • Service has endpoints but traffic does not reach the backend pods
  • Intermittent connection failures that resolve after kube-proxy restart

ClusterIP reachability depends entirely on kube-proxy syncing the correct iptables/IPVS rules. If kube-proxy lags or crashes, all services on that node are affected.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Endpoints for the service

    kubectl get endpoints <svc> -n <namespace>

  • kube-proxy mode and pod health

    kubectl get pods -n kube-system -l k8s-app=kube-proxy

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • kube-proxy logs show iptables-restore error
  • Service has endpoints but ClusterIP does not route

Confirm before fixing

kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50 | grep -iE 'error|iptables'
READ

Check kube-proxy logs

Fix

kubectl rollout restart daemonset/kube-proxy -n kube-system
CHANGE

Restart kube-proxy to re-sync iptables rules

Expected: daemonset.apps/kube-proxy restarted

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get endpoints <svc> -n <namespace>
READ

Confirm endpoints exist

2
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50
READ

Check kube-proxy logs for errors

3
kubectl rollout restart daemonset/kube-proxy -n kube-system
CHANGE

Restart kube-proxy to re-sync rules

4
ssh <node> dmesg | grep conntrack
READ

Check for conntrack table full

Prevention

  • Monitor kube-proxy pod health and iptables sync latency.
  • Raise net.netfilter.nf_conntrack_max on high-connection nodes.
  • Use IPVS mode on clusters with many services to avoid iptables performance issues.
  • Alert on pod-to-pod connection failures using synthetic probes.

Related runbooks

Same stack, same failure family — worth a skim.

conntrack table full -- nf_conntrack: table full, dropping packetcritical

dmesg shows 'nf_conntrack: table full, dropping packet'

NetworkingAdvancedUSE3 causes
Open runbook
DNS resolution failing inside pods — CoreDNS issuescritical

curl: Could not resolve host inside pods

KubernetesIntermediate2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.