criticalProductionUSE#etcd#apiserver#control-plane#latency

etcd latency high / apiserver healthz failing — cluster-wide timeouts

kubectl commands return context deadline exceeded or connection refused

Likely source:availabilityperformancedata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • kubectl commands return context deadline exceeded or connection refused
  • apiserver /healthz returns non-OK or times out
  • All operations across the cluster are slow or failing

etcd is the single most critical component in a Kubernetes control plane. Disk latency > 10ms on the etcd data directory is the most common root cause of cluster-wide failures.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • apiserver healthz

    curl -sk https://<apiserver>:6443/healthz

  • etcd member status

    ETCDCTL_API=3 etcdctl endpoint health --endpoints=https://127.0.0.1:2379

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • etcd disk fsync duration > 100ms
  • apiserver healthz slow or failing

Confirm before fixing

ssh <control-plane> iostat -x 1 3
READ

Measure disk latency

Expected: await < 10ms for etcd disk

Fix

ssh <control-plane> sudo systemctl restart etcd
DESTRUCTIVE

Restart etcd after moving to a faster disk (SSD/NVMe required for etcd)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -sk https://<apiserver>:6443/healthz
READ

Check apiserver health

2
ETCDCTL_API=3 etcdctl endpoint health --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
READ

Check etcd health

3
ssh <control-plane> iostat -x 1 3
READ

Measure disk latency

4
ssh <control-plane> sudo systemctl restart etcd
DESTRUCTIVE

Restart etcd after disk fix

Prevention

  • Use SSD or NVMe disks for etcd data directory (never HDD).
  • Monitor etcd disk fsync duration and alert at > 50ms.
  • Run etcd on dedicated control-plane nodes with no other workloads.
  • Backup etcd regularly and test restore procedures.

Related runbooks

Same stack, same failure family — worth a skim.

Worker node NotReady — node down or kubelet failurecritical

kubectl get nodes shows NotReady for one or more workers

KubernetesAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.