Load Balancing
Intermediate 12 min read Load Balancing

Target Groups and Health Checks

The fleet and the probe that watches it

Target groups define the fleet a balancer routes to; health checks are its eyes on that fleet. Correct health-check design (path, thresholds, interval) prevents draining live traffic to a half-dead box.

target grouphealth checkbackend poolthresholddraining
Prerequisites: Load Balancing Basics
Before we start

What you will learn

  • Design a target group + health-check profile
  • Tune interval/thresholds to match app behavior
  • Explain draining and deregistration delays
In 60 seconds

A target group (AWS) / backend pool (Azure, GCP) holds instances behind a listener. Health checks probe an endpoint on your app; failures trip thresholds that remove targets from rotation and trigger replacement flows in autoscaling.

Introduction

What is this?

AWS target groups: protocol+port, health-check settings (Path, IntervalSeconds, TimeoutSeconds, Healthy/UnhealthyThreshold). Azure backend pools: LB probes / App Gateway probes (path, interval, timeout, thresholds). GCP: backend services with instance group members + health checks (HTTP/HTTPS/TCP). Drain: deregistration delay (AWS), drain-mode, graceful shutdown — lets in-flight requests finish before a target is removed.

Motivation

Why do you need it?

  • Removing a flaky target prevents latency incidents before customers feel them.
  • Autoscaling + health checks self-heal a fleet without tickets.
  • Tuning thresholds avoids flapping (too eager) or zombie targets (too lazy).
  • Draining makes deploys and failures graceful.
Think about it

Mental model

Imagine

A concierge checking which receptionists are at their desks (and not on a call) before sending guests

Think of

Health checks + draining

If a receptionist is away (health check fails), guests don't get sent there; when it's shift change (deploy), in-progress conversations (requests) are finished before their desk shuts.

Mechanics

How it works

  • The balancer probes the health path on the health port every interval (default 30s).
  • Success threshold on consecutive passes marks 'healthy'; failure threshold marks 'unhealthy'.
  • Unhealthy targets leave rotation; the balancer returns to them only after recovery threshold.
  • Draining/deregistration keeps existing connections alive for a delay while blocking new ones.
  • Autoscaling uses the same health signal to terminate-and-replace groups.

Health-check lifecycle.

Probe
Pass x2
Fail x3
Drain
Recover
Hands-on

Examples

A sane health-check config

A sane health-check config
# AWS: target group health check tuned for slow cold-start apps
aws elbv2 create-target-group --name web-tg --protocol HTTP \
  --port 8080 --vpc-id vpc-12345 \
  --health-check-protocol HTTP --health-check-path /healthz \
  --health-check-interval-seconds 15 --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 --unhealthy-threshold-count 3 \
  --deregistration-delay-timeout-seconds 60

15s interval, 2/3 thresholds, 60s drain: enough to avoid flapping while still catching dead targets quickly.

Real world

In production

  • An ALB health check paths /internal/ready that also validates the DB connection pool (fail fast).
  • App Gateway probes catch an app that accepts TCP but serves no HTTP (both checked).
  • GCP backend with instance group + customized health check routes around a degrading shard.
Guidelines

Best practices

  • Make the health endpoint cheap and deep enough: verify app liveness, not just OS liveness.
  • Set interval ≥ app cold-start time and thresholds that tolerate loader spikes.
  • Always set a drain/deregistration delay on services with in-flight work.
  • Exclude health endpoints from access logs or rate them; they hammer every interval.
Watch out

Common mistakes

  • Health path that requires auth → balancer can't authenticate → flapping.
  • Thresholds too eager: a 500ms blip removes a target, causing cascading load shifts.
  • Zero drain delay on a service that processes jobs — mid-request kills.
  • Probing TCP-only and missing an app that listens but never serves 200s.
Root cause analysis

Common issues

Error
Targets test healthy via curl but balancer says unhealthy
Root cause

Probe port/interval mismatch or SG blocks the health port

Fix

Match probe port + protocol exactly; allow the health port from the LB SG.

Error
Fleet flaps in waves
Root cause

Interval too short vs app startup or a shared dependency outage

Fix

Lengthen interval/thresholds and check the shared dependency's health.

When things break

Debugging tips

  • Use describe-target-health to see per-target state and reasons (e.g., 'unhealthy timeout').
  • Curl the health path from the target's own host to separate app vs network issues.
  • If targets flap together, the app depends on a shared downstream (DB) — check that first.
  • Watch metrics: 'healthy host count' dropping during a release = deploy red flag.
Interviews

Interview checkpoint

Practice

Hands-on lab

Design and Stress a Health-Check Profile
25 min · intermediate
In progress
Objectives
  • Configure a target group with tuned health-check params
  • Simulate threshold behavior with a small Python model
  • Verify drain keeps in-flight requests safe

Design and Stress a Health-Check Profile

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/1
1

The balancer probes the health endpoint on the same port the app serves.

Go deeper

Official documentation

AWS — Target groups
docs
Azure — Backend pools and probes
docs
GCP — Health checks
docs