System Design
Beginner 12 min read System Design

Scalability Basics

Scalability is a system's ability to keep working well as load grows. Vertical scaling buys a bigger machine; horizontal scaling adds more machines. The key to horizontal scale is statelessness — if any instance can serve any request, you can add and remove instances freely.

scalabilityvertical scalinghorizontal scalingstatelessautoscalingload testing
Prerequisites: Git Basics & Workflow Network Fundamentals
Introduction

What is this?

Scalability measures how a system handles growing workload — more users, more requests, more data — without degrading latency or availability. Vertical scaling adds CPU, memory and disk to a single machine, which is simple but has hard physical and cloud instance-size limits. Horizontal scaling runs the workload across many machines and is the strategy behind every large consumer platform, because it can grow (and shrink) without replacing hardware.

Motivation

Why do you need it?

  • Traffic is not steady: launch days, viral posts and sales events can multiply load 10-100x in minutes.
  • Vertical scaling hits a ceiling — the biggest cloud instance has finite CPU, memory and I/O bandwidth.
  • Horizontal scaling grows capacity linearly: double the instances, roughly double the throughput.
  • Stateless horizontal scale gives you elasticity — scale up during a spike and back down after, saving cost.
  • Distributing across machines also spreads risk: one failed server no longer takes down the whole service.
Think about it

Mental model

Imagine

A single checkout counter at a grocery store

Think of

Many checkout lanes with a traffic-directing manager

One counter can only serve one customer at a time; hiring a faster cashier (vertical) helps only up to a point. Adding lanes (horizontal) lets many customers check out in parallel — but now you need a manager pointing people to the emptiest lane. That manager is your load balancer.

Mechanics

How it works

  • Requests enter through a single entry point — a DNS name or a load balancer — that hides the pool of servers behind it.
  • Application instances are kept stateless: session data and caches live in shared stores (Redis, a database), so any instance can serve any request.
  • A load balancer distributes each request to an instance using round-robin, least-connections or hashing.
  • An autoscaler watches metrics (CPU, request latency, queue depth) and adds or removes instances to match demand.
  • Data stores scale separately because state is the hard part: reads go to replicas, and only the data tier holds durable state.

From one server to an elastic pool of stateless instances behind a load balancer.

Users
Load balancer
Instance pool
Autoscaler
Shared state
Response
Reference

Key commands

docker compose up -d --scale api=3

Start three stateless API replicas behind the compose service name.

Network created. Container api-1, api-2, api-3 Started.
docker compose ps

List the running instances and their status.

api-1 Up 1 minute (healthy)  0.0.0.0:8080->8080/tcp
curl -s http://localhost:8080/health

Verify one replica is healthy.

{"status":"ok"}
ab -n 5000 -c 100 http://localhost:8080/

ApacheBench: 5,000 requests with 100 concurrent connections.

Requests per second: 842.13 [#/sec] (mean)
docker compose logs --tail=20 api

Tail logs across all replicas to spot errors or uneven load.

nproc

Print the CPU count — the vertical resource you are buying when you scale up.

8
Hands-on

Examples

A containerized API you can scale horizontally

A containerized API you can scale horizontally
# docker-compose.yml
services:
  api:
    image: myapp/api:1.4
    environment:
      - DATABASE_URL=postgres://db:5432/app
      - REDIS_URL=redis://cache:6379
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 3s
      retries: 3

The API reads its configuration from environment variables and has a health endpoint, so it is portable and stateless. Once the image is built, you run `docker compose up -d --scale api=3` to start three instances and the compose network load-balances between them. Every instance connects to the same Postgres and Redis, so any of the three can handle any request.

Real world

In production

  • Streaming platforms run the stateless web tier behind an autoscaler driven by CPU utilization and request latency, adding instances during live events and removing them after.
  • E-commerce platforms pre-scale to several times baseline before a flash sale, then let the autoscaler shrink the pool back down over the next hour.
  • Batch and image-processing workers scale on queue depth: each worker pulls a job, processes it, and exits, so you can run hundreds of workers without worrying about routing.
  • Multi-region deployments replicate the same stateless tier in each region so users are served by the nearest pool, with only the data layer needing cross-region care.
Guidelines

Best practices

  • Keep application instances stateless: sessions, locks and temp files belong in shared stores or object storage, never on instance disk.
  • Make instances disposable — a new instance must be able to boot, register and serve without manual setup.
  • Define a health check that reflects real readiness (DB connection, migrations applied), not just process liveness.
  • Instrument request latency, error rate and queue depth before scaling; scale on those, not on raw CPU alone.
  • Use autoscaling with warm-up and cool-down windows so you do not churn instances during a burst.
Watch out

Common mistakes

  • Scaling the wrong tier: adding API servers while the shared database or a lock service remains the actual bottleneck.
  • Storing sessions in memory, then discovering that horizontal scaling randomly logs users out.
  • Making instances stateful (uploaded files on disk, in-process caches) so a new instance behaves differently from the rest.
  • Sizing for the peak and paying for it all year instead of using elasticity to scale down when traffic is quiet.
Root cause analysis

Common issues

Error
OutOfMemoryError / Node.js 'heap out of memory' after scaling up
Root cause

Each instance now handles more concurrent requests, or a leak compounds across more instances.

Fix

Add memory limits and connection pooling, tune the runtime heap, and inspect heap dumps for the leak.

Error
p95 latency spiked right after adding instances
Root cause

All new instances cold-started at once and hammered the database, or warmed caches all expired together.

Fix

Pre-warm caches, stagger instance startup, and ramp traffic gradually instead of a full cutover.

Error
All traffic is hitting one instance
Root cause

Sticky sessions or a hash that is not randomized after scale-out.

Fix

Remove session affinity or rebalance on instance join; keep instances stateless so routing can be free.

When things break

Debugging tips

  • Check the load balancer's access log for the instance ID behind each request to spot a single hot instance.
  • Compare p50, p95 and p99: if p95 is fine but p99 explodes, look for GC pauses, cold cache misses or tail latency in a shared store.
  • Watch the request queue depth at the entry point — if it grows while instance CPU is idle, the bottleneck is downstream, not the app tier.
  • Correlate the deployment timestamp with the latency change; many 'scale' problems are actually bad-release problems.
Interviews

Interview checkpoint

Practice

Hands-on lab

Scale a Stateless App from 1 to 5 Instances
30 min · beginner
In progress
Objectives
  • Package a stateless API as a container
  • Run one instance and verify health
  • Scale the pool to five replicas behind the compose network
  • Observe how in-process state breaks horizontal scaling

Scale a Stateless App from 1 to 5 Instances

30 min · beginner

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which of the following is an example of horizontal scaling?

2

A stateless application can serve a request from any instance in the pool.

3

A sales page goes viral and CPU on your single API server is pegged at 100%. You want minimal change and fast relief tonight. What is the best first step?

Quick reference

Cheat sheet

Scaling vocabulary
Vertical scalingBigger single machine; simple, bounded
Horizontal scalingMore machines; needs statelessness
Stateless instanceNo session/disk state; any instance serves any request
AutoscalingAdd/remove instances from metrics
ElasticityScale up during spikes and back down to save cost
Quick checks
docker compose up -d --scale api=NScale a compose service to N replicas
ab -n 5000 -c 100 URLQuick load test of a stateless tier
docker compose psConfirm how many replicas are actually up
docker compose logs --tail=50 apiCheck errors and request distribution
Go deeper

Official documentation

AWS Auto Scaling — What is Auto Scaling?
docs
Kubernetes Horizontal Pod Autoscaler
docs
Azure Architecture Center — Autoscaling best practices
docs
NGINX HTTP Load Balancing
docs