Lesson 8 of 8 in Networking for DevOps

Debugging network problems layer by layer

Core10 min

Network debugging is the habit of testing each layer a request depends on in a fixed order, so the first failing check names the broken layer. The order is name resolution, reachability, port, TLS and then the HTTP response.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: DNS: how names become addresses, HTTP, HTTPS and TLS certificates, Firewalls and security groups

The problem this solves

A message like it cannot connect hides at least five different failures. Guessing, or changing three settings at once, leaves you unsure which change mattered and which one broke something else.

Each layer has a command that answers one question. Asking them in order turns a vague outage into a single failing check you can act on.

How it works

The order to test a failing connection
The order to test a failing connectionStart from the symptom and check in order whether the name resolves, the address is reachable, the port is open, the certificate is valid and the HTTP response is right.1 resolves?2 reachable?3 port open?4 cert valid?5 rightresponse?Symptomit cannot connectNamedigReach IPping, traceroutePort opennc, ssTLSopenssl s_clientHTTP answercurl -v
  • People and clients
  • Network
  • Security
  • Runs code

Start from the symptom and check in order whether the name resolves, the address is reachable, the port is open, the certificate is valid and the HTTP response is right.

  1. Symptom to Name: 1 resolves?
  2. Name to Reach IP: 2 reachable?
  3. Reach IP to Port open: 3 port open?
  4. Port open to TLS: 4 cert valid?
  5. TLS to HTTP answer: 5 right response?

Move through the boxes in order and stop at the first failure. If the name does not resolve, nothing after it can work, so fix DNS first. If the name resolves and the address does not respond, look at routing and firewalls. If the address is reachable and the port is closed, look at the service and its bind address.

Two answers matter most. A refusal comes back immediately and means the host is there and nothing listens on that port. A timeout means silence, which points to a drop by a firewall or a route that does not exist. A reset in the middle of a request means the other side, or a proxy in between, closed the connection.

Test from the same place the failing client runs. A laptop, a server in another subnet and a container have different resolvers, routes and rules, so a check from the wrong vantage point proves nothing about the real problem.

curl can also tell you which layer is slow. Its -w option prints the time spent on name lookup, connecting, TLS and the full request, so a slow service stops being a guess.

Try it yourself

Run these against a public site first, then repeat them with your own service as the target. They only read and connect.

  1. 1
    Name: does it resolve?
    dig +short example.com

    No output or a NXDOMAIN status means stop here and fix DNS.

  2. 2
    Route: is the address reachable?
    ping -c 3 example.com
    traceroute example.com

    Some networks block ICMP, so a failing ping is a hint and does not prove a fault. traceroute may need installing, and tracepath is an alternative on Linux. Stars in the output are hops that do not answer.

  3. 3
    Port: can you open a connection?
    nc -vz -w 3 example.com 443

    -z connects and exits, -w 3 sets a 3 second wait. Success means the port accepts connections. Refused and timed out mean different things, see the refusal and timeout steps in the firewalls concept.

  4. 4
    TLS: is the certificate good?
    openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | grep 'Verify return code'

    Verify return code: 0 (ok) means the chain and name checks passed.

  5. 5
    HTTP: is the answer what you expect, and where is the time?
    curl -s -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total} status=%{http_code}\n' https://example.com

    Each time is cumulative from the start. A large jump between two values shows which layer is slow.

  6. 6
    See the whole conversation when something is odd
    curl -v https://example.com -o /dev/null

    The output lists name lookup, connection, the TLS handshake and the request and response headers in order.

What breaks, and where to look

  • ping fails but the website works, or ping works and the site does not load.

    WHY IT HAPPENS
    Ping uses ICMP, and a firewall or security group can block it while allowing TCP port 443. The reverse also happens, where ICMP is allowed and the web port is blocked.
    CHECK FIRST
    Test the real protocol on the real port with nc -vz -w 3 <host> <port> or curl -v, and use ping only as a hint.
  • It works from your laptop and fails from the server or the container, with Could not resolve host or a timeout.

    WHY IT HAPPENS
    The failing machine has a different resolver, route, proxy setting or egress rule. You tested from the wrong place.
    CHECK FIRST
    Run the same checks from there, for example docker exec <container> sh -c 'getent hosts <name>' or an SSH session on the server. Compare /etc/resolv.conf and ip route on both.
  • A connection works for a while and then breaks with Connection reset by peer, or only large requests fail.

    WHY IT HAPPENS
    The other end or a proxy in the middle closed the connection, for example on an idle or request-size limit, or the service crashed mid-request.
    CHECK FIRST
    Read the logs of the proxy and the service around the time of the failure, and use curl -v to see how far the request got.

When not to use this

  • These tools show that something fails and where. They do not show why a service inside is slow or wrong. Move on to the application logs and metrics once the network checks pass.
  • For deep problems, such as retransmits or odd resets, capturing packets with tcpdump or Wireshark shows what is on the wire. Learn the layer-by-layer checks first, and capture packets when they stop explaining the fault.

Check yourself

Why test in a fixed order?

Each layer depends on the one before it, so the first failing check identifies the broken layer and stops you guessing.

How do a refusal and a timeout differ in what they point at?

A refusal means the host answered and nothing listens on the port. A timeout means silence, which suggests a firewall drop or a missing route.

Why test from the same place as the failing client?

Resolvers, routes and firewall rules differ between machines and containers, so a pass on your laptop says nothing about the failing environment.

Lesson 23 of 92 in the DevOps path

Search

Search cheat sheets and projects