highAdvancedRED#rds#replica#lag#read-scaling

RDS read replica lagging behind the primary

Readers return stale data relative to the primary

Likely source:performancedata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

Readers return stale data relative to the primary
ReplicaLag CloudWatch metric climbs or stays elevated
Many concurrent replicas behind after a write spike

ReplicaLag is seconds that translate directly to stale reads. Route 'read-your-writes' traffic to the primary, and treat any sustained >60s lag as an availability defect, not a metric quirk.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

ReplicaLag metric
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name ReplicaLag --dimensions Name=DBInstanceIdentifier,Value=<REPLICA-ID> --start-time <ISO> --end-time <ISO> --period 300 --statistics Maximum
Replica instance context
aws rds describe-db-instances --db-instance-identifier <REPLICA-ID> --query 'DBInstances[0].{Class:DBInstanceClass,Storage:AllocatedStorage,MultiAZ:MultiAZ,IOPS:Iops}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

Replica CPU or ReadIOPS pinned at max during lag
Lag grows proportionally to primary write rate

Confirm before fixing

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization --dimensions Name=DBInstanceIdentifier,Value=<REPLICA-ID> --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 300 --statistics Maximum

READ

Replica saturation

Fix

aws rds modify-db-instance --db-instance-identifier <REPLICA-ID> --db-instance-class <LARGER-CLASS> --apply-immediately

CHANGE

Right-size the replica to the replay rate

Expected: Modifying to larger class

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name ReplicaLag --dimensions Name=DBInstanceIdentifier,Value=<REPLICA-ID> --start-time $(date -u -d '-6 hours' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 300 --statistics Maximum

READ

Baseline the lag curve

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization --dimensions Name=DBInstanceIdentifier,Value=<REPLICA-ID> --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 300 --statistics Average

READ

Is the replica saturated or idle?

aws rds modify-db-instance --db-instance-identifier <REPLICA-ID> --db-instance-class <LARGER-CLASS> --apply-immediately

CHANGE

Right-size if saturated

aws ssm start-session --target <BASTION-INSTANCE-ID>

READ

From SQL, confirm excludes/DB engines: check for in-flight long transactions as the alternate path