highAdvancedUSE#cloud sql#replica#lag#replication#read only

Cloud SQL read replica lagging behind primary

Reads from the replica return stale data

Likely source:availabilityperformancedata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Reads from the replica return stale data
  • Cloud SQL metrics show replication_lag_seconds climbing
  • Replica status shows REPLICATION_LAG or not APPLYING
  • Replica instance enters an error/suspended state

A replica that permanently lags is serving stale data silently — acceptable for some analytics, dangerous for user-facing reads. Scale the replica before you reason about schema fixes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Replica lag metric

    gcloud monitoring time-series list --filter='metric.type="cloudsql.googleapis.com/database/replication/lag"' --format=json

  • Replica status

    gcloud sql instances describe <REPLICA> --format='value(settings.availabilityType,serviceAccountEmailAddress,ipAddresses)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Lag spikes coincide with batch jobs / large writes on the primary
  • Replica CPU is saturated applying changes
  • Lag stays high even with a lean read load

Confirm before fixing

gcloud sql instances describe <REPLICA> --format='value(settings.tier,settings.dataDiskSizeGb,ipAddresses)'
READ

Check the replica's compute tier and disk vs primary

Expected: Replica tier significantly smaller than the primary

Fix

gcloud sql instances patch <REPLICA> --tier=db-custom-<CPU>-<MEMORY_MB>
CHANGE

Scale the replica to match or exceed the primary's tier

Expected: Updated [<REPLICA>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud monitoring time-series list --filter='metric.type="cloudsql.googleapis.com/database/replication/lag"' --format=json
READ

Confirm lag is real and its trend

2
gcloud sql instances patch <REPLICA> --tier=db-custom-<CPU>-<MEMORY_MB>
CHANGE

Scale the replica if it is underpowered

3
SELECT pid, now()-xact_start AS dur, query FROM pg_stat_activity WHERE state <> 'idle' ORDER BY dur DESC LIMIT 5;
READ

Look for a long-running transaction on the primary

4
gcloud sql instances describe <REPLICA> --format='value(settings.activationPolicy)'
READ

Confirm the replica is in ALWAYS-active state, not paused

Prevention

  • Right-size replicas to at least the primary's CPU for write-heavy primaries.
  • Monitor replication lag with an alert above a business-tolerable threshold.
  • Chunk large batch writes on the primary to keep replay bursts small.
  • Keep indexes in sync between primary and replicas via migration tooling.

Related runbooks

Same stack, same failure family — worth a skim.

Cloud SQL connection limit hit — too many connectionshigh

App logs show 'too many connections' from the database driver

Google CloudIntermediateUSE3 causes
Open runbook
Cloud SQL storage auto-grow failed — disk fullcritical

gcloud sql instances describe shows dataDiskSizeGb at capacity

Google CloudIntermediateUSE3 causes
Open runbook
GCE persistent disk stuck in DETACHING statehigh

gcloud compute instances describe shows a disk in state DETACHING

Google CloudAdvancedUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.