Lesson 7 of 8 in Networking for DevOps
Load balancers and reverse proxies
A reverse proxy accepts requests on behalf of one or more backend servers and forwards them. A load balancer is a reverse proxy that spreads requests across several identical backends and stops sending to ones that fail their health checks.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: HTTP, HTTPS and TLS certificates, Ports, TCP and UDP
The problem this solves
One server is a single point of failure and has a capacity limit. If you run several copies, clients need a single address to use and something that keeps traffic away from a copy that has died.
You also want one place to handle certificates, redirect HTTP to HTTPS, and send /api to one service and everything else to another.
How it works
- People and clients
- Network
- Runs code
- Control
A client sends each request to the load balancer, which picks one healthy backend, and a health check keeps failing backends out of the pool.
- Client to Load balancer: 1 request
- Load balancer to Backend A: 2 forward to one
- Load balancer to Backend B: 2 or to the other
- Load balancer to Health check: probes backends
- People and clients
- Network
- Runs code
The proxy reads the request path and sends it to the API service or to the web service.
- Client to Reverse proxy: 1 request
- Reverse proxy to API service: 2 path starts /api
- Reverse proxy to Web service: 2 otherwise
Clients only know the proxy's name and address. The proxy keeps a pool of backends and chooses one for each request or connection. Round robin goes in turn, and least connections prefers the backend with the fewest open ones. Health checks, such as requesting /health every few seconds, take a backend out of the pool when it stops answering correctly.
A layer 4 balancer forwards TCP or UDP connections without reading them. A layer 7 balancer, which includes nginx and cloud application load balancers, reads the HTTP request. That lets it route by hostname or path, add headers and, with a certificate, end TLS so that backends can speak plain HTTP.
The backend no longer sees the client directly. It sees the proxy's address, so the proxy usually passes the original client address in a header such as X-Forwarded-For, and the application has to read it from there.
The proxy itself becomes a single point of failure, which is why cloud providers run load balancers across several machines and zones.
Try it yourself
Two Python servers play the backends and nginx in Docker plays the load balancer. You need Docker and Python 3, and all of it runs on your laptop for free. Use three terminals, and run everything from ~/net-lab.
- 1Create two backends that identify themselves
mkdir -p lb/a lb/b && echo A > lb/a/index.html && echo B > lb/b/index.html python3 -m http.server 8001 --directory lb/a
Run the
python3line in terminal 1. In terminal 2 runpython3 -m http.server 8002 --directory lb/bfrom the same folder. - 2Write the nginx config
cat > nginx.conf <<'EOF' events {} http { upstream app { server host.docker.internal:8001; server host.docker.internal:8002; } server { listen 80; location / { proxy_pass http://app; proxy_set_header X-Forwarded-For $remote_addr; } } } EOFThe
upstreamblock is the pool. Nginx uses round robin unless you say otherwise. - 3Start nginx as the load balancer (terminal 3)
docker run --rm -p 8080:80 --add-host=host.docker.internal:host-gateway -v "$PWD/nginx.conf:/etc/nginx/nginx.conf:ro" nginx:stable
--add-hostlets the container reach your laptop on Linux. Docker Desktop on macOS and Windows already provides the name. - 4Send requests and watch them alternate
for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/; done
Expect the letters A and B to alternate. Each request gets a new backend.
- 5Kill one backend and look again
for i in 1 2 3 4; do curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/; donePress Ctrl+C in terminal 2 first. Nginx tries the dead backend, gets a refusal, and retries the request on the other one, so the status stays 200. Then stop everything with Ctrl+C and delete
~/net-lab/lband~/net-lab/nginx.conf.
What breaks, and where to look
502 Bad Gatewayfrom the proxy.- WHY IT HAPPENS
- The proxy could not get a valid answer from the backend. It is down, on a different port, or refused the connection. The nginx error log says
connect() failed (111: Connection refused) while connecting to upstream. - CHECK FIRST
- Read the proxy's error log (
docker logs <container-id>here), then runcurl http://<backend-address>:<port>/from the proxy host to test the backend directly.
504 Gateway Timeoutafter a long wait.- WHY IT HAPPENS
- The backend accepted the connection and did not answer before the proxy's timeout ran out. The app is slow, stuck on a database call or overloaded.
- CHECK FIRST
- Time the backend directly with
curl -w '%{time_total}\n' -o /dev/null -s <backend-url>and look at the app's logs for slow requests. Raise the proxy timeout only after you know why it is slow.
Every request in the application log comes from the same IP, the proxy's, and rate limits or geo rules misbehave.
- WHY IT HAPPENS
- The backend sees the proxy as the client. The real address only exists in the
X-Forwarded-Forheader, which the application has to be told to trust. - CHECK FIRST
- Print the request headers in the app, then enable its trusted-proxy setting so it reads the forwarded address only from your proxy.
When not to use this
- A load balancer shares load across copies that behave the same. State kept in one server's memory, such as sessions, needs a shared store or sticky routing, or users get logged out between requests.
- A single small service does not need its own proxy tier. Add one when you have several backends, need TLS in one place, or must route by path or host.
Check yourself
What is the difference between a layer 4 and a layer 7 balancer?
Layer 4 forwards connections without reading them. Layer 7 reads HTTP, so it can route by host or path and terminate TLS.
What does a health check do?
It probes each backend regularly and removes one from the pool when it stops answering correctly.
Why do backends need X-Forwarded-For?
The proxy makes the connection to them, so the backend sees only the proxy's address. The header carries the real client address.
Keep going
Lesson 22 of 92 in the DevOps path