Lesson 7 of 8 in Networking for DevOps

Load balancers and reverse proxies

Advanced9 min

A reverse proxy accepts requests on behalf of one or more backend servers and forwards them. A load balancer is a reverse proxy that spreads requests across several identical backends and stops sending to ones that fail their health checks.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: HTTP, HTTPS and TLS certificates, Ports, TCP and UDP

The problem this solves

One server is a single point of failure and has a capacity limit. If you run several copies, clients need a single address to use and something that keeps traffic away from a copy that has died.

You also want one place to handle certificates, redirect HTTP to HTTPS, and send /api to one service and everything else to another.

How it works

A load balancer shares requests across healthy backends
A load balancer shares requests across healthy backendsA client sends each request to the load balancer, which picks one healthy backend, and a health check keeps failing backends out of the pool.1 request2 forward to one2 or to the otherprobes backendsClientone address to useLoad balancerpicks a backendBackend Aapp copyBackend Bapp copyHealth checkGET /health
  • People and clients
  • Network
  • Runs code
  • Control

A client sends each request to the load balancer, which picks one healthy backend, and a health check keeps failing backends out of the pool.

  1. Client to Load balancer: 1 request
  2. Load balancer to Backend A: 2 forward to one
  3. Load balancer to Backend B: 2 or to the other
  4. Load balancer to Health check: probes backends
A reverse proxy routes by path
A reverse proxy routes by pathThe proxy reads the request path and sends it to the API service or to the web service.1 request2 path starts /api2 otherwiseClientGET /api/usersReverse proxyreads the pathAPI servicehandles /apiWeb servicehandles everything else
  • People and clients
  • Network
  • Runs code

The proxy reads the request path and sends it to the API service or to the web service.

  1. Client to Reverse proxy: 1 request
  2. Reverse proxy to API service: 2 path starts /api
  3. Reverse proxy to Web service: 2 otherwise

Clients only know the proxy's name and address. The proxy keeps a pool of backends and chooses one for each request or connection. Round robin goes in turn, and least connections prefers the backend with the fewest open ones. Health checks, such as requesting /health every few seconds, take a backend out of the pool when it stops answering correctly.

A layer 4 balancer forwards TCP or UDP connections without reading them. A layer 7 balancer, which includes nginx and cloud application load balancers, reads the HTTP request. That lets it route by hostname or path, add headers and, with a certificate, end TLS so that backends can speak plain HTTP.

The backend no longer sees the client directly. It sees the proxy's address, so the proxy usually passes the original client address in a header such as X-Forwarded-For, and the application has to read it from there.

The proxy itself becomes a single point of failure, which is why cloud providers run load balancers across several machines and zones.

Try it yourself

Two Python servers play the backends and nginx in Docker plays the load balancer. You need Docker and Python 3, and all of it runs on your laptop for free. Use three terminals, and run everything from ~/net-lab.

  1. 1
    Create two backends that identify themselves
    mkdir -p lb/a lb/b && echo A > lb/a/index.html && echo B > lb/b/index.html
    python3 -m http.server 8001 --directory lb/a

    Run the python3 line in terminal 1. In terminal 2 run python3 -m http.server 8002 --directory lb/b from the same folder.

  2. 2
    Write the nginx config
    cat > nginx.conf <<'EOF'
    events {}
    http {
      upstream app {
        server host.docker.internal:8001;
        server host.docker.internal:8002;
      }
      server {
        listen 80;
        location / {
          proxy_pass http://app;
          proxy_set_header X-Forwarded-For $remote_addr;
        }
      }
    }
    EOF

    The upstream block is the pool. Nginx uses round robin unless you say otherwise.

  3. 3
    Start nginx as the load balancer (terminal 3)
    docker run --rm -p 8080:80 --add-host=host.docker.internal:host-gateway -v "$PWD/nginx.conf:/etc/nginx/nginx.conf:ro" nginx:stable

    --add-host lets the container reach your laptop on Linux. Docker Desktop on macOS and Windows already provides the name.

  4. 4
    Send requests and watch them alternate
    for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/; done

    Expect the letters A and B to alternate. Each request gets a new backend.

  5. 5
    Kill one backend and look again
    for i in 1 2 3 4; do curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/; done

    Press Ctrl+C in terminal 2 first. Nginx tries the dead backend, gets a refusal, and retries the request on the other one, so the status stays 200. Then stop everything with Ctrl+C and delete ~/net-lab/lb and ~/net-lab/nginx.conf.

What breaks, and where to look

  • 502 Bad Gateway from the proxy.

    WHY IT HAPPENS
    The proxy could not get a valid answer from the backend. It is down, on a different port, or refused the connection. The nginx error log says connect() failed (111: Connection refused) while connecting to upstream.
    CHECK FIRST
    Read the proxy's error log (docker logs <container-id> here), then run curl http://<backend-address>:<port>/ from the proxy host to test the backend directly.
  • 504 Gateway Timeout after a long wait.

    WHY IT HAPPENS
    The backend accepted the connection and did not answer before the proxy's timeout ran out. The app is slow, stuck on a database call or overloaded.
    CHECK FIRST
    Time the backend directly with curl -w '%{time_total}\n' -o /dev/null -s <backend-url> and look at the app's logs for slow requests. Raise the proxy timeout only after you know why it is slow.
  • Every request in the application log comes from the same IP, the proxy's, and rate limits or geo rules misbehave.

    WHY IT HAPPENS
    The backend sees the proxy as the client. The real address only exists in the X-Forwarded-For header, which the application has to be told to trust.
    CHECK FIRST
    Print the request headers in the app, then enable its trusted-proxy setting so it reads the forwarded address only from your proxy.

When not to use this

  • A load balancer shares load across copies that behave the same. State kept in one server's memory, such as sessions, needs a shared store or sticky routing, or users get logged out between requests.
  • A single small service does not need its own proxy tier. Add one when you have several backends, need TLS in one place, or must route by path or host.

Check yourself

What is the difference between a layer 4 and a layer 7 balancer?

Layer 4 forwards connections without reading them. Layer 7 reads HTTP, so it can route by host or path and terminate TLS.

What does a health check do?

It probes each backend regularly and removes one from the pool when it stops answering correctly.

Why do backends need X-Forwarded-For?

The proxy makes the connection to them, so the backend sees only the proxy's address. The header carries the real client address.

Lesson 22 of 92 in the DevOps path

Search

Search cheat sheets and projects