Lesson 4 of 8 in Networking for DevOps

HTTP, HTTPS and TLS certificates

Core10 min

HTTP is the request and response protocol the web and most APIs use. HTTPS is HTTP carried inside TLS, which encrypts the traffic and uses a certificate to prove which server you are talking to.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Ports, TCP and UDP, DNS: how names become addresses

The problem this solves

Almost every service you deploy speaks HTTP, and almost every outage report arrives as a status code or a certificate error. If you cannot read a request, a status code and a certificate, you are guessing.

On a shared network, plain HTTP can be read and altered by anyone along the path. TLS fixes that, and it introduces new failures, such as expired certificates, that look like the site is broken.

How it works

What happens when a client opens an HTTPS connection
What happens when a client opens an HTTPS connectionThe client connects and says hello, the server presents a certificate, the client checks the signer against its trusted list, and then encrypted requests flow.1 hello, 4 encrypted data2 shows certificate3 check the signerClientbrowser or curlServerport 443Certificatename, expiry, signerTrusted CAslist on the client
  • People and clients
  • Runs code
  • Security

The client connects and says hello, the server presents a certificate, the client checks the signer against its trusted list, and then encrypted requests flow.

  1. Client to Server: 1 hello, 4 encrypted data
  2. Server to Certificate: 2 shows certificate
  3. Client to Trusted CAs: 3 check the signer

An HTTP request has a method such as GET or POST, a path, headers and sometimes a body. The response has a status code, headers and a body. The Host header names the site, which lets many sites share one IP address. HTTP/2 and HTTP/3 change how the bytes travel and keep the same meaning.

Status codes come in families. 2xx means success. 3xx means go somewhere else, such as 301 for a permanent redirect. 4xx means the request was wrong, for example 404 not found or 401 and 403 for authentication and permission problems. 5xx means the server or something behind it failed, for example 502 or 503.

TLS starts after TCP connects. The server presents a certificate that names the hosts it is valid for, has an expiry date, and is signed by a certificate authority, or CA. The client checks that the name matches, the dates are valid, and the signature chains to a CA it already trusts. Then both sides agree on keys and encrypt everything that follows.

TLS protects the contents of the conversation from being read or changed in transit, and it tells you that the server holds a certificate for that name. It does not make the server honest, and the destination IP address and usually the hostname are still visible to the network.

Try it yourself

Start a local server in ~/net-lab for the first steps (python3 -m http.server 8000 --bind 127.0.0.1). The last step reads a public certificate and needs internet access.

  1. 1
    See the raw request and response
    curl -v http://127.0.0.1:8000/ -o /dev/null

    Lines starting with > are what curl sent, and lines starting with < are what came back, beginning with the status line.

  2. 2
    Produce a 200 and a 404
    curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/
    curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/no-such-page

    You should see

    200
    404
  3. 3
    Fetch only the headers of a real HTTPS site
    curl -sI https://example.com

    The first line is the status. The headers below it describe the response.

  4. 4
    Read the certificate a server presents
    openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates

    You get the subject (who it names), the issuer (the CA that signed it), and notBefore and notAfter, the validity window. -servername sends the hostname so the server picks the right certificate.

What breaks, and where to look

  • curl: (60) SSL certificate problem: certificate has expired or unable to get local issuer certificate.

    WHY IT HAPPENS
    The certificate is past its notAfter date, the server did not send its intermediate certificate, or a proxy on your network re-signs traffic with a CA your machine does not trust.
    CHECK FIRST
    Run the openssl s_client command from this page against the host and look at Verify return code, then print the dates with openssl x509 -noout -dates. Fix the certificate. Do not make -k your fix.
  • curl: (60) SSL: no alternative certificate subject name matches target host name '10.0.1.20'.

    WHY IT HAPPENS
    The certificate is valid for names such as app.example.com, and you connected by IP address or by another name that it does not list.
    CHECK FIRST
    Connect with the hostname on the certificate. To test before DNS exists, use curl --resolve app.example.com:443:10.0.1.20 https://app.example.com.
  • curl http://site prints nothing or an empty page, while the browser shows the site.

    WHY IT HAPPENS
    The server answered with a redirect, such as 301 to the HTTPS address, and curl does not follow redirects unless asked.
    CHECK FIRST
    Run curl -sI http://site and read the status and Location header, then use curl -L to follow it.

When not to use this

  • Do not turn off certificate checking (-k, verify=False) to get past an error. That removes the one thing that proves who you are talking to. Fix the trust or the certificate.
  • TLS only protects data in transit. It does nothing for a vulnerable app, a stolen password or data stored unencrypted on a disk.

Check yourself

What do 4xx and 5xx status codes tell you about where to look?

4xx means the request itself was wrong or not permitted. 5xx means the server or something behind it failed.

What three things does a client check on a certificate?

That the name matches the host, that today is inside the validity dates, and that it chains to a CA the client trusts.

What does TLS not hide?

The destination IP address and usually the hostname. It also does not prove that the server is trustworthy.

Lesson 19 of 92 in the DevOps path

Search

Search cheat sheets and projects