Lesson 7 of 7 in Security basics for DevOps
TLS certificates: what they prove and how they expire
TLS encrypts traffic between a client and a server, and a certificate is how the server proves it owns the name you connected to. A certificate names the host, is signed by an issuer your client already trusts, and is only valid between two dates.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Shift left and think like a defender
The problem this solves
Encryption alone is not enough. If you encrypt a conversation with whoever answers, you may be talking privately to an impostor. A client needs to be sure that the server it reached is the one that owns the name in the address bar or the URL in its config.
For DevOps engineers the practical pain is expiry and mismatch. Certificates have an end date, so one that nobody renewed turns into an outage at an awkward hour, and a certificate issued for the wrong name makes every client refuse the connection. You need to read a certificate quickly and know which field to look at.
How it works
- People and clients
- Runs code
- Security
The client connects, the server presents its certificate, the client checks it against its list of trusted issuers, and only then sends encrypted traffic.
- Client to Server: 1 connect
- Server to Certificate: 2 presents
- Certificate to Trusted issuers: 3 checked against
- Trusted issuers to Client: 4 trusted: encrypt
A certificate is a small signed document with three things you will inspect most. The subject names, which on modern certificates live in the Subject Alternative Name (SAN) field, say which hostnames it is valid for. The issuer says who signed it. The validity dates say when it starts and stops being accepted.
Trust works by chain. Your operating system or browser ships with a list of certificate authorities (CAs) it trusts. The server's certificate is signed by an intermediate CA, which is signed by one of the trusted roots. The client walks that chain and checks each signature, the names and the dates.
A certificate you generate yourself is self-signed, so nobody's trust list contains it. That is fine for local experiments, and it is why clients refuse it until you tell them explicitly to trust that one certificate. The steps below do exactly that on your own machine.
Try it yourself
You need openssl (version 1.1.1 or newer) and curl. Everything stays on your machine except the last step, which only reads a public site's certificate. On Windows use WSL or Git Bash. In Git Bash, set MSYS_NO_PATHCONV=1 before the -subj command.
- 1Create a throwaway self-signed certificate
mkdir -p ~/tls-demo && cd ~/tls-demo openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 1 \ -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost"
This makes a private key and a certificate valid for one day for the name
localhost. They are for this demo only. - 2Read the fields that matter
openssl x509 -in cert.pem -noout -subject -issuer -dates -ext subjectAltName
Subject and issuer are the same, which is what self-signed means.
notAfteris the expiry. The SAN list is the set of names this certificate is valid for. - 3Check expiry the way a monitor would
openssl x509 -in cert.pem -noout -checkend 3600; echo "exit code: $?" openssl x509 -in cert.pem -noout -checkend 172800; echo "exit code: $?"
-checkend Nasks whether the certificate will still be valid N seconds from now. The first call says it will not expire within an hour and exits 0. The second asks about two days, which is past this certificate's one day, so it says it will expire and exits 1. Monitoring scripts use that exit code. - 4Serve it and watch trust fail, then succeed
openssl s_server -accept 8443 -cert cert.pem -key key.pem -www & sleep 1 curl -sS https://localhost:8443/ | head -n 3 curl -sS --cacert cert.pem https://localhost:8443/ | head -n 3 kill %1
The first
curlis refused because no trusted issuer signed the certificate. The second tells curl to trust exactly this file and receives a response. The error wording for the first one depends on your curl and TLS library, but it will mention the certificate or issuer not being trusted. - 5Inspect a real site's certificate
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \ | openssl x509 -noout -subject -issuer -dates
This only reads the public certificate.
-servernamesends the hostname so the server returns the right certificate. Notice that the issuer differs from the subject, and note thenotAfterdate. Remove the demo files withrm -r ~/tls-demo.
What breaks, and where to look
curl: (60) SSL certificate problem: certificate has expiredor a browser warning about an expired certificate.- WHY IT HAPPENS
- The certificate's
notAfterdate has passed and nobody renewed it, or an automatic renewal job failed quietly. - CHECK FIRST
- Run
openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -dates. Then check the renewal job's logs and add an expiry alert well before the date.
curl: (60) SSL certificate problem: unable to get local issuer certificate- WHY IT HAPPENS
- The client cannot build a chain to a trusted root. The server often forgot to send its intermediate certificate, or the client's trust list is missing or old.
- CHECK FIRST
- Run
openssl s_client -connect <host>:443 -servername <host> -showcerts </dev/nulland count how many certificates the server sends. A leaf alone without its intermediate is the usual culprit.
no alternative certificate subject name matches target host namewhen you connect.- WHY IT HAPPENS
- The name you used is not in the certificate's SAN list. This happens when you connect by IP address or by a different alias than the certificate was issued for.
- CHECK FIRST
- Print the names with
openssl x509 -in cert.pem -noout -ext subjectAltNameor inspect the live one as above, then connect using a name that is listed.
When not to use this
- Never turn off verification (for example
curl -korverify=False) to make an error go away outside a throwaway test. It removes the one check that proves who you are talking to. - Self-signed certificates are fine for local learning. For anything others use, get certificates from a public CA or your organisation's internal CA, and automate renewal.
Check yourself
What three things on a certificate do you check first?
The names it is valid for (SAN), the issuer that signed it, and the start and end dates.
Why does a client reject a self-signed certificate?
No certificate authority in the client's trust list signed it, so the client cannot confirm who issued it.
What does openssl x509 -checkend 3600 tell you?
Whether the certificate will still be valid in 3600 seconds. Exit code 0 means yes and 1 means it will have expired.
Keep going
Lesson 92 of 92 in the DevOps path