Compute
Intermediate 14 min read Compute

Autoscaling

Fleets that match demand

Autoscaling grows or shrinks instance fleets to match demand: target tracking keeps average CPU sensible; scheduled scaling covers known peaks; and the scaling process itself fails in characteristic ways you must design around.

autoscalingASGVMSSMIGscale outscale intarget tracking
Prerequisites: Virtual Machines and Instances
Before we start

What you will learn

  • Explain scale out/in triggers and cool-down/cooldown semantics
  • Combine target tracking, scheduled and predictive scaling
  • Design ASG/VMSS/MIG that survive scale-in races and capacity gaps
In 60 seconds

A group (ASG, VMSS, MIG) watches metrics like average CPU or request count, adds instances when utilization climbs and removes them when idle — using launch templates and health checks so the fleet always matches demand.

Introduction

What is this?

AWS Auto Scaling Groups: launch template + min/max/desired, scaling policies (target tracking, step, scheduled, predictive), health checks (EC2 and ELB), lifecycle hooks and termination policies. Azure VMSS: autoscale rules on metrics/CPU, schedules, scaling actions, health probes. GCP Managed Instance Groups: autoscaler with target utilization, CPU/requests metrics, cool-down and buffer sizing.

Motivation

Why do you need it?

  • Pay for demand, not peak: fleets shrink at night automatically.
  • Smoothly absorb traffic spikes that would tip one big box.
  • Self-healing: replace unhealthy instances without human contact.
  • Predictable costs: keep min warm, scale elastically above it.
Think about it

Mental model

Imagine

A supermarket opening and closing checkout lanes

Think of

Autoscaling

Manager watches queue length (metric), opens lanes when lines grow (scale out), closes them when quiet (scale in), and always keeps enough lanes for the schedule — the same trigger/policy relation as autoscaling.

Mechanics

How it works

  • Group definition: min, max, desired, launch template (image + size + user-data) that every new instance uses.
  • Target tracking scales to a threshold: average CPU 60% adds/removes instances to hold that average.
  • Scale-out is fast (instances start eagerly); scale-in is conservative and drains over cooldown so spot price and traffic settle.
  • Health checks flag unhealthy instances; the group terminates and replaces them.
  • Scheduled scaling pre-warms for known peaks (9am login wave) and lifecycle hooks pause termination for cleanup.

Autoscaling control loop.

Metrics
Policy
Scale decision
Launch/drain
Fleet updated
Hands-on

Examples

Target-tracking autoscaling in each cloud

Target-tracking autoscaling in each cloud
# AWS: target tracking on average CPU at 60%
aws autoscaling put-scaling-policy --auto-scaling-group-name web   --policy-name cpu60 --policy-type TargetTrackingScaling   --target-tracking-configuration '{"PredefinedMetricSpecification":  {"PredefinedMetricType":"ASGAverageCPUUtilization"},"TargetValue":60.0}'
# Azure VMSS: autoscale rule file (CPU > 70% add, < 30% remove)
az monitor autoscale create -g app-rg --resource vmss-1   --min-count 2 --max-count 10 --count 2
# GCP MIG autoscaler
gcloud compute instance-groups managed set-autoscaling web-mig   --max-num-replicas 10 --target-cpu-utilization 0.6

All three use target-metric autoscaling policies over instance groups.

Real world

In production

  • A checkout service scales on ALB target response time with a 15-instance cap.
  • ETL workers scale by SQS queue depth via a custom metric policy.
  • A VMSS scales on custom in-guest CPU counters with scheduled surge before payroll runs.
Guidelines

Best practices

  • Set a min that sustains base traffic and a max that bounds cost.
  • Add warm pools/pre-warm for bursty latency-sensitive tiers.
  • Make instances killable: no state outside external stores (S3/DB/queue).
  • Test scale-in: rolling termination should not drop in-flight requests.
Watch out

Common mistakes

  • Autoscaling over a single oversized instance — peak-loaded and expensive.
  • Ignoring cooldown: policies fight each other and 'thrash' the fleet.
  • State lives on the instance (local logs, sessions) → scale-in drops data.
  • Max set too low: scale-out can never catch the spike (looks like 'not working').
Root cause analysis

Common issues

Error
Group never scales above 2 instances
Root cause

Max too low or policy throttled

Fix

Raise max, verify the policy evaluates the right metric for the right group.

Error
New instances fail health checks, group replaces constantly
Root cause

Health check port misconfigured or boot script failing

Fix

Fix the launch template/health check; check cloud-init for boot errors.

When things break

Debugging tips

  • Check the scaling activity history: the group logs reasons for each add/remove.
  • Scale-out slow? Check cloud-init and image boot time; optimize with warm pools.
  • Cooldown + step policy interactions cause oscillation — log policy evaluations.
  • If the group never scales in, suspend/resume processes and inspect the termination policy.
Interviews

Interview checkpoint

Practice

Hands-on lab

Design and Simulate an Autoscaling Policy
30 min · intermediate
In progress
Objectives
  • Translate traffic requirements into min/max/desired and targets
  • Simulate the control loop with a Python model
  • Design a throttling and cooldown that avoids oscillation

Design and Simulate an Autoscaling Policy

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/1
1

What prevents an autoscaler from oscillating between add and remove?

Go deeper

Official documentation

AWS — Auto Scaling groups
docs
Azure — VMSS autoscale
docs
GCP — Autoscaling groups
docs