Cloud Architecture
Advanced 16 min read Cloud Architecture

Scaling & Performance Design

The load will come — design the on-ramp

Design for load before the load arrives: horizontal scaling (stateless tiers, LB, autoscale), performance levers (caching, queries, batching, CDN), and capacity planning that keeps p99 sane. Scale-testing + autoscale policy tuning complete the loop.

scalingautoscalecachingCDNbatchingcapacityp99load test
Prerequisites: Compute Load Balancing
Before we start

What you will learn

  • Apply horizontal-scaling levers to tiers
  • Design performance (cache/query/batch/CDN) at each layer
  • Plan capacity + autoscale thresholds with headroom
In 60 seconds

Scaling = add replicas horizontally (stateless) + LB/autoscale, not bigger boxes. Performance = cache hot reads, tune the queries, batch the calls, edge the static. Capacity planning = measure, load-test, tune autoscale so the spike is absorbed, not survived as drama.

Introduction

What is this?

Horizontal scaling: stateless tiers (the enabler), LB/autoscale, partitioned queues/work are partitioned for scale; DB scaling via replicas/partitioning (data tier usually the bottleneck). Performance levers: caching (hot reads at each layer — edge CDN, service cache, DB cache), query/index tuning + mviews, batching (fewer calls/messages), async offload, connection pooling, compression. Autosacle tuning: cooldown, min/max, proactive (schedule ahead of known peaks), scale metrics (CPU, latency, queue depth). Capacity planning: load testing (k6/Gatling/Artillery + provider LB logs), peak model, instance-size selection with headroom, and budget: the autoscale policy IS the capacity commitment. Anti-pattern: 'add a bigger box' as design — scale horizontally unless state bars it. p99 gates: define p99 per tier; measure after scale events.

Motivation

Why do you need it?

  • Horizontal > vertical: replicas give you failure isolation + elasticity + blue/green deploy room.
  • Performance levers (cache/query/batch/CDN) multiply capacity before you buy any scale.
  • Autoscale done right absorbs spikes; done wrong (cold scale) turns a spike into an incident.
  • Capacity planning + load test makes 'the spike' a rehearsed event, not a prayer.
Think about it

Mental model

Imagine

A highway widening (vertical) vs. building an express lane + tolling (horizontal + cache/batch)

Think of

Scale + performance

Upgrading to a 20-lane highway is the expensive last resort; adding lane-splitting (cache), carpool lanes (batch), and express toll routes (CDN for static) moves more traffic per dollar. And you plan the road for the holiday rush, not after it.

Mechanics

How it works

  • Keep tiers stateless so replicas are free teleportation.
  • Add LB/autoscale with sane thresholds + schedule ahead of known peaks.
  • Apply performance levers per layer (CDN → cache → query/batch).
  • Scale-test: reach the target p99, find the elastic bottleneck (usually DB), fix it.
  • Tune + document the capacity plan; re-run after traffic changes.

Scale design.

Stateless tiers
Autoscale + LB
Perf levers
Load test
Capacity plan
Hands-on

Examples

Autoscale policy idea

Autoscale policy idea
# ASG policy: web tier
#  min 4, max 32; CPU threshold add at 65% (cooldown 300s)
#  proactive: scale to 24 at 09:00 before weekday peak (scheduled action)
#  scale-down: wait 15m below 35%
# Database: read replicas +2 when replica CPU > 70%
# p99 gate after any scale event: < 180ms (else → cache/query work first).

A policy with floor/cap, scheduled anticipation, and a p99 gate — autoscaling as designed capacity, not an emergency brake.

Real world

In production

  • Web tier: ASG + ALB, CPU-based + scheduled pre-scale for known peaks.
  • Cache hot reads: CDN for static + service cache for reads (Redis/ElastiCache/mv view).
  • Load test quarterly with current peak replay: catch the new bottleneck before the new season.
Guidelines

Best practices

  • Prefer horizontal + stateless everywhere state law allows; vertical only where it must.
  • Autoscale with cooldown + schedule (pre-scale known peaks), not pure panic-add.
  • Put performance levers BEFORE scale: cache/query/batch/CDN buy the multiplier.
  • Data tier: replicas/partitioning planned, not 'we'll upgrade the box'.
  • p99 gates measured; load test tied to capacity, markets, and prod-like shape.
Watch out

Common mistakes

  • Vertical-only 'bigger box' design that caps out and dies painfully (single point at peak).
  • Autoscale with cold instances (startup slow) → the spike is an incident before scale finishes.
  • Scaling compute but the DB is single-write — every replica waits on one write path.
  • Load testing on a happy-path approximation, so the DB bottleneck hides.
When things break

Debugging tips

  • Scale completes but p99 still blows → the bottleneck is not compute (DB/cache/lock); measure per-tier p99.
  • Autoscale waits forever → scale thrash/cooldown misconfigured or the metric lags (CPU sees the load late).
  • Spike hits while scaling up slowly → cold-start/launch image slow; pre-scale by schedule or warm pools.
  • DB write path queueing → partition/shard the write path or batch; replicas only fix reads.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The enabler that makes horizontal scaling trivial:

2

Why load-test before the season, not during:

Go deeper

Official documentation

AWS autoscaling docs
docs
Azure autoscale + performance
docs
GCP autoscaler docs
docs