highIntermediateRED#cloud functions#429#resource exhausted#concurrency#scaling

Cloud Functions 429 RESOURCE_EXHAUSTED / concurrency pressure

Function returns 429 RESOURCE_EXHAUSTED under load

Likely source:availabilityscaling

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Function returns 429 RESOURCE_EXHAUSTED under load
  • HTTP API responses show 'Quota exceeded for quota metric requests'
  • Gen2 errors report function instance limit reached (concurrency cap)
  • Spikes in traffic cause a jump in 429s before any autoscaling catches up

429 RESOURCE_EXHAUSTED on Cloud Functions is either a request-rate quota or an instance-concurrency ceiling. Read whichever metric the error body names; raising the wrong one does not help.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Error response body

    curl -i <FUNCTION_URL>

  • Rate/quota metrics

    gcloud monitoring time-series list --filter='metric.type="cloudfunctions.googleapis.com/function/instance_count"' --format=json

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • 429 body names requests/day or requests/100 seconds metric
  • Traffic spike correlates exactly with the 429 window
  • Rate is below the region resource quota but above function defaults

Confirm before fixing

curl -s -o /dev/null -w '%{http_code}' '<FUNCTION_URL>' && gcloud services quota list --consumer <PROJECT> --service cloudfunctions.googleapis.com --format=json
READ

Reproduce and read the Cloud Functions service quota

Expected: A quota metric at or near its limit

Fix

gcloud services consumer quota update <PROJECT> --service=cloudfunctions.googleapis.com --metric=cloudfunctions.googleapis.com/quota/requests --unit=1/min/{project} --limit=1000
CHANGE

Request a higher per-project request quota

Expected: Quota updated

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -i '<FUNCTION_URL>'
READ

Confirm the 429 and read its quota message

2
gcloud monitoring time-series list --filter='metric.type="cloudfunctions.googleapis.com/function/instance_count"' --format=json
READ

Check whether instances are pinned at max

3
gcloud services consumer quota update <PROJECT> --service=cloudfunctions.googleapis.com --metric=<QUOTA_METRIC> --unit=<UNIT> --limit=<NEW_LIMIT>
CHANGE

Raise the limiting quota metric

Prevention

  • Raise function instance max + request quota deliberately before launch seasons.
  • Use min-instances warmers for latency-and-429-sensitive functions.
  • Add client-side retry with backoff for 429 responses.
  • Monitor 429 rate per function as a scaling signal.

Related runbooks

Same stack, same failure family — worth a skim.

Cloud Functions timeout exceeded (300s max)medium

Function logs show Function execution took too long, finished with status: 500

Google CloudIntermediateRED3 causes
Open runbook
High CPU throttling — containers starved by CPU limitsmedium

High response latency with low CPU saturation on the node

KubernetesAdvancedUSE2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.