Lesson 4 of 8 in Networking for DevOps
HTTP, HTTPS and TLS certificates
HTTP is the request and response protocol the web and most APIs use. HTTPS is HTTP carried inside TLS, which encrypts the traffic and uses a certificate to prove which server you are talking to.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Ports, TCP and UDP, DNS: how names become addresses
The problem this solves
Almost every service you deploy speaks HTTP, and almost every outage report arrives as a status code or a certificate error. If you cannot read a request, a status code and a certificate, you are guessing.
On a shared network, plain HTTP can be read and altered by anyone along the path. TLS fixes that, and it introduces new failures, such as expired certificates, that look like the site is broken.
How it works
- People and clients
- Runs code
- Security
The client connects and says hello, the server presents a certificate, the client checks the signer against its trusted list, and then encrypted requests flow.
- Client to Server: 1 hello, 4 encrypted data
- Server to Certificate: 2 shows certificate
- Client to Trusted CAs: 3 check the signer
An HTTP request has a method such as GET or POST, a path, headers and sometimes a body. The response has a status code, headers and a body. The Host header names the site, which lets many sites share one IP address. HTTP/2 and HTTP/3 change how the bytes travel and keep the same meaning.
Status codes come in families. 2xx means success. 3xx means go somewhere else, such as 301 for a permanent redirect. 4xx means the request was wrong, for example 404 not found or 401 and 403 for authentication and permission problems. 5xx means the server or something behind it failed, for example 502 or 503.
TLS starts after TCP connects. The server presents a certificate that names the hosts it is valid for, has an expiry date, and is signed by a certificate authority, or CA. The client checks that the name matches, the dates are valid, and the signature chains to a CA it already trusts. Then both sides agree on keys and encrypt everything that follows.
TLS protects the contents of the conversation from being read or changed in transit, and it tells you that the server holds a certificate for that name. It does not make the server honest, and the destination IP address and usually the hostname are still visible to the network.
Try it yourself
Start a local server in ~/net-lab for the first steps (python3 -m http.server 8000 --bind 127.0.0.1). The last step reads a public certificate and needs internet access.
- 1See the raw request and response
curl -v http://127.0.0.1:8000/ -o /dev/null
Lines starting with
>are what curl sent, and lines starting with<are what came back, beginning with the status line. - 2Produce a 200 and a 404
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/no-such-pageYou should see
200 404
- 3Fetch only the headers of a real HTTPS site
curl -sI https://example.com
The first line is the status. The headers below it describe the response.
- 4Read the certificate a server presents
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates
You get the subject (who it names), the issuer (the CA that signed it), and
notBeforeandnotAfter, the validity window.-servernamesends the hostname so the server picks the right certificate.
What breaks, and where to look
curl: (60) SSL certificate problem: certificate has expiredorunable to get local issuer certificate.- WHY IT HAPPENS
- The certificate is past its
notAfterdate, the server did not send its intermediate certificate, or a proxy on your network re-signs traffic with a CA your machine does not trust. - CHECK FIRST
- Run the
openssl s_clientcommand from this page against the host and look atVerify return code, then print the dates withopenssl x509 -noout -dates. Fix the certificate. Do not make-kyour fix.
curl: (60) SSL: no alternative certificate subject name matches target host name '10.0.1.20'.- WHY IT HAPPENS
- The certificate is valid for names such as
app.example.com, and you connected by IP address or by another name that it does not list. - CHECK FIRST
- Connect with the hostname on the certificate. To test before DNS exists, use
curl --resolve app.example.com:443:10.0.1.20 https://app.example.com.
curl http://siteprints nothing or an empty page, while the browser shows the site.- WHY IT HAPPENS
- The server answered with a redirect, such as
301to the HTTPS address, and curl does not follow redirects unless asked. - CHECK FIRST
- Run
curl -sI http://siteand read the status andLocationheader, then usecurl -Lto follow it.
When not to use this
- Do not turn off certificate checking (
-k,verify=False) to get past an error. That removes the one thing that proves who you are talking to. Fix the trust or the certificate. - TLS only protects data in transit. It does nothing for a vulnerable app, a stolen password or data stored unencrypted on a disk.
Check yourself
What do 4xx and 5xx status codes tell you about where to look?
4xx means the request itself was wrong or not permitted. 5xx means the server or something behind it failed.
What three things does a client check on a certificate?
That the name matches the host, that today is inside the validity dates, and that it chains to a CA the client trusts.
What does TLS not hide?
The destination IP address and usually the hostname. It also does not prove that the server is trustworthy.
Keep going
Lesson 19 of 92 in the DevOps path