System Design
Intermediate 15 min read System Design

Load Balancing

A load balancer is the single entry point in front of a pool of servers that distributes traffic, runs health checks and fails over dead instances. Layer-4 balances TCP connections; layer-7 understands HTTP and routes by URL, header or host. Choosing the right algorithm and health check is what separates an even, resilient pool from a stack that collapses on one hot server.

load balancerL4/L7round robinleast connectionshealth checkssticky sessionsnginxfailover
Prerequisites: Scalability Basics Network Fundamentals HTTP & the Web
Introduction

What is this?

A load balancer sits in front of a group of servers and presents a single virtual IP or DNS name to clients. It distributes each connection or request to a backend, removes unhealthy servers from rotation via health checks, and gives you safe rollouts through connection draining. Layer-4 balancers work on IP and TCP port; layer-7 balancers parse HTTP and can route by path, host or header, terminate TLS and rewrite requests.

Motivation

Why do you need it?

  • Without balancing, one server absorbs traffic, saturates, and the tail latency of the whole site degrades.
  • Health checks let the pool self-heal: a crashing server is ejected before clients ever see a 500.
  • Deployments become safe: drain connections on the old version while new instances warm up, then cut over.
  • SSL termination at the edge offloads cryptography from app servers and centralizes certificate management.
  • Sticky routing (hashing) keeps a user on one server for the duration of a request or session when needed.
Think about it

Mental model

Imagine

Airport check-in with one long queue

Think of

A supervisor sending each passenger to the shortest desk

The supervisor (load balancer) keeps every agent busy, reassigns customers when an agent calls in sick (health check), and stops sending people to a closed desk (failover). Without the supervisor, everyone funnels to the nearest desk and that line grows forever.

Mechanics

How it works

  • Clients resolve one virtual IP or hostname; all traffic lands on the load balancer, not individual servers.
  • L4 balancers forward packets or TCP connections based on destination IP and port without inspecting payloads.
  • L7 balancers terminate the connection, inspect the HTTP request (path, host, headers) and proxy it to a chosen backend.
  • An algorithm — round-robin, least-connections, IP hash, consistent hash — picks the target for each unit of work.
  • Health checks (TCP, HTTP or scripted) run on an interval and take unhealthy servers out of the active set.
  • During deployments, connection draining lets in-flight requests finish before an instance is removed.

Traffic flows through one virtual IP, is picked by an algorithm, and dead servers are pruned by health checks.

Clients
Load balancer
Algorithm
Health checks
Backend pool
Response
Reference

Key commands

docker compose up -d --scale api=3

Start a three-instance backend pool to balance across.

Container api-1, api-2, api-3 Started.
nginx -t

Validate the nginx configuration file before reloading.

nginx: configuration file /etc/nginx/nginx.conf test is successful
nginx -s reload

Apply the new upstream configuration without dropping connections.

for i in $(seq 1 20); do curl -s http://localhost:8080/; echo; done

Hit the balancer repeatedly and watch the instance identifier alternate between backends.

ab -n 5000 -c 100 http://localhost:8080/

Load-test the balanced pool and compare throughput with the single-instance run.

Requests per second: 2140.44 [#/sec] (mean)
docker logs nginx-lb --since 5m

Inspect the balancer access and error logs for upstream failures.

Hands-on

Examples

NGINX layer-7 upstream with least-connections and failover

NGINX layer-7 upstream with least-connections and failover
# nginx.conf - HTTP load balancer
upstream backend {
    least_conn;
    server api-1.internal:8080 max_fails=3 fail_timeout=30s;
    server api-2.internal:8080 max_fails=3 fail_timeout=30s;
    server api-3.internal:8080 backup;
}
 
server {
    listen 80;
 
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_connect_timeout 2s;
    }
}

The upstream block names the backend pool. least_conn routes each request to the server with the fewest active connections, which suits long-lived or uneven requests. max_fails and fail_timeout mark a server down after three failures in 30 seconds, and proxy_next_upstream retries on the next server if one errors. The third server is a backup, used only when both primary servers are down.

Real world

In production

  • Cloud load balancers (AWS ALB/ELB, Google Cloud LB) front every public service, terminating TLS and splitting traffic across availability zones so a zone failure does not take the service down.
  • Database read replicas sit behind an L4 balancer so analytics and reporting queries spread across replicas while writes keep going to the leader.
  • Global DNS-based balancing (GSLB) routes users to the nearest region; within each region, an L7 balancer splits traffic by URL pattern.
  • Service meshes and gRPC systems balance at the connection level with round-robin over a resolver, adding retries and circuit breaking at the client side.
Guidelines

Best practices

  • Health checks should probe the application, not just the TCP port: return 200 only when the app can actually serve requests.
  • Prefer least-connections (or consistent hashing) over plain round-robin when requests have very different durations.
  • Run the balancer as an active/passive or active/active pair — a single load balancer is a single point of failure.
  • Enable connection draining and disable stickiness during deploys so in-flight requests finish on the old version.
  • Pass X-Forwarded-For and the scheme header so backends see the real client IP and preserve HTTPS redirects.
Watch out

Common mistakes

  • Using sticky sessions as a default, which pins users to instances and defeats horizontal scaling.
  • Round-robin across heterogeneous instances, overloading small instances and underusing large ones.
  • Shallow health checks — a health endpoint that returns 200 even when the database is down.
  • Adding servers to the pool without draining old ones, dropping in-flight requests during every deploy.
Root cause analysis

Common issues

Error
502 Bad Gateway on every request
Root cause

All upstream servers are failing health checks, or the upstream server block is empty.

Fix

Check each backend's health endpoint, nginx error log, and nginx -t; mark the servers reachable before reloading.

Error
All traffic lands on one backend despite round-robin
Root cause

Client-side keepalive or IP-hash routing pins connections to one server, or the other servers are marked down.

Fix

Remove sticky/hash config, confirm the other backends pass health checks, and re-run the distribution test.

Error
High latency at the balancer even though backends are idle
Root cause

No keepalive to upstreams, so every request opens a fresh TCP handshake, or the balancer is in a remote region.

Fix

Enable proxy_http_version 1.1 and a keepalive connection pool to upstreams; place the balancer near clients and backends.

When things break

Debugging tips

  • Read the balancer access log: it records which backend handled each request, exposing a single hot server.
  • Bypass the balancer with curl --resolve example.com:80:backend-ip to test one backend directly.
  • Request repeatedly and check the distribution of backend identifiers to verify the algorithm is actually spreading load.
  • Compare total time to upstream_time in logs: a large gap means queueing at the balancer or the backend.
Interviews

Interview checkpoint

Practice

Hands-on lab

Balance Traffic Across a Node Pool with NGINX
40 min · intermediate
In progress
Objectives
  • Run a three-instance Node API pool that reports its identity
  • Configure NGINX as an L7 load balancer in front of it
  • Verify requests are distributed and observe failover
  • Apply least-connections and health checks to harden the pool

Balance Traffic Across a Node Pool with NGINX

40 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which load-balancing layer can route requests by URL path or Host header?

2

A plain TCP health check is always sufficient to know whether a backend is healthy.

3

Users must land on the same backend server for the duration of a single session. Which routing mechanism is most appropriate?

Quick reference

Cheat sheet

Algorithms
Round-robinEven sequential distribution; ignores load
Least-connectionsRoute to fewest active connections
IP hashSame client -> same server; loses affinity on changes
Consistent hashingStable mapping with minimal reshuffling
WeightedProportional to instance size/capacity
NGINX quick config
upstream backend { server a; server b; }Define the pool
least_conn;Choose the algorithm
server x max_fails=3 fail_timeout=30s;Failover on repeated errors
proxy_next_upstream error timeout http_502;Retry another server on failure
proxy_set_header X-Real-IP $remote_addr;Pass the real client IP
Go deeper

Official documentation

NGINX HTTP Load Balancing
docs
AWS Application Load Balancer
docs
HAProxy Documentation
docs
Envoy Proxy — What is Envoy?
docs