Load Balancing
Intermediate 13 min read Load Balancing

Autoscaling and Traffic Distribution

Balancer and fleet growing together

Load balancers and autoscaling are designed to work as one unit: the fleet scales on LB-side signals (request count, latency), the LB follows fleet size, and weighted/least-connections distribution keeps load balanced across capacities.

autoscalingtraffic distributionweighted routingtarget trackingrequest count
Prerequisites: Autoscaling
Before we start

What you will learn

  • Integrate autoscaling groups with the balancer's target group
  • Design scaling on request-based metrics
  • Explain weighted and least-connection distributions
In 60 seconds

Autoscaling keeps the fleet sized to demand and registers/removes instances with the LB automatically. Scales on 'requests per target' or load-balancer latency; the LB spins distribution across whatever the group produces, weighted where speeds differ.

Introduction

What is this?

Integration: ASG/VMSS/MIG attach a target group; scale events register/deregister instances. Scale metrics: requests-per-target on ALB, queue depth via custom metrics; App Gateway autoscale on RU; GCP MIG autoscaler on utilization. Distribution: round-robin, least-outstanding-requests, weighted targets (AWS target weight, Azure weight-based backends, GCP capacity scaler/weights). Weighted routing is used at canary releases (0/90/10) and when instance capabilities differ.

Motivation

Why do you need it?

  • LBs + autoscaling together = the fleet matches demand AND is load-balanced.
  • LB-side metrics capture the actual service experience better than host CPU alone.
  • Weighted distribution enables canary/traffic-splitting without DNS tricks.
  • Least-requests prevents a slow box from hogging the queue.
Think about it

Mental model

Imagine

A mall opening more checkout lanes (fleet) and a manager balancing queues (LB)

Think of

Autoscaling + traffic distribution

The manager opens lanes when lines get long (ASG on queue depth), and the registration desk (LB) sends each customer to whichever lane is shortest, weighting lanes if some workers are faster.

Mechanics

How it works

  • ASG attaches a target group; instances launched by the group register automatically.
  • Scaling policies read metrics like ALB 'RequestCountPerTarget' or CPU and adjust the fleet.
  • Scale-in terminates instances from the group; the LB drains them before actual termination.
  • Weighted targets split traffic by weight for canary; least-outstanding-requests avoids stragglers.
  • The LB DNS/endpoint stays constant — only the fleet behind it changes.

LB + ASG loop.

Requests
Metric
Scale policy
ASG + LB sync
Fleet balanced
Hands-on

Examples

Scale on request-based metrics

Scale on request-based metrics
# ALB-backed ASG with target tracking on requests per target
aws autoscaling attach-load-balancer-target-groups \
  --auto-scaling-group-name web-asg --target-group-arns <tg>
aws autoscaling put-scaling-policy --auto-scaling-group-name web-asg \
  --policy-name req-target --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/<lb>/<tg>"
    }, "TargetValue": 1000}'

The group scales when the average request load per target crosses the target — a service-level signal, not host CPU.

Real world

In production

  • A retail back office scales 2→40 nodes on ALB request count during a flash sale.
  • GCP MIG autoscaling hooks utilization to the LB so pushes never saturate any zone.
  • Azure App Gateway RU autoscale sizes gateways as the fleet grows underneath.
Guidelines

Best practices

  • Scale on the metric that predicts add-times: request count beats CPU-for-latency services.
  • Couple ASG and LB lifecycle: attach the target group, let the group drive registrations.
  • Weighted canaries: 0/90/10 then 50/50 then 100 — with LB drain for the old version.
  • Set max group size to bound scale-related spend and LB connection burst.
Watch out

Common mistakes

  • Scaling on CPU while the real signal is request latency → wrong fleet sizes.
  • Manual register/deregister with AMI changes — group+LB should own the lifecycle.
  • Weighted 0 on a canary that never gets verified (0% receives a trickle — that's intentional).
  • Autoscaling on LB metrics while the LB is the bottleneck (scale LB first).
Root cause analysis

Common issues

Error
Group connects but 0 traffic
Root cause

Listener/target-group association or weight 0

Fix

Confirm the listener forwards to the TG and the correct rule priority matches.

Error
Scale in terminates a bursty new instance
Root cause

Group drained it while the spike returned

Fix

Use cooldown/warm pool, and choose an instance protection policy for recent launches.

When things break

Debugging tips

  • 'Scaling did nothing' — check the metric exists and the ASG is attached to the TG.
  • Requests peaked but group flat: policy throttled or max reached; inspect scale activity history.
  • Canary at 10% drifting: verify weight actually applies (listener rule → target group weight).
  • If instances spin up but immediately unhealthy, re-check the health-check port on the new AMI.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/1
1

Why prefer request-count-per-target over host CPU as a scaling metric?

Go deeper

Official documentation

AWS — Scaling based on LB metrics
docs
Azure — Autoscale App Gateway
docs
GCP — MIG autoscaling
docs