APIs
Intermediate 14 min read APIs

Rate Limits & Retry Strategies

Dance with the limiter, don't fight it

Provider rate limits and token-rate caps determine throughput and reliability. Exponential backoff with jitter, honoring Retry-After, and distinguishing retryable from non-retryable failures are core skills.

rate limits429retrybackoffjitterRetry-Afterthroughput
Prerequisites: HTTP & REST for AI
Before we start

What you will learn

  • Read rate-limit headers and respect Retry-After
  • Implement exponential backoff with jitter
  • Classify failures as retryable or non-retryable
Introduction

What is this?

Model providers enforce rate limits — requests per minute and/or tokens per minute — enforced with 429 responses. Reliable AI integrations treat limits as a resource: they pace requests under the ceiling, honor Retry-After, and retry transient failures with exponential backoff plus jitter. Naive retries (immediate re-send, or infinite retries) make limits worse and hang pipelines.

Motivation

Why do you need it?

  • 429s are often a sign you're at the limit, not a bug — design for them.
  • Backoff without jitter causes synchronized thundering-herd retry storms.
  • Retrying a 4xx validation error wastes calls and never succeeds.
  • Token-per-minute limits gate cost and utilization; understanding them prevents silent throttling and overage bills.
Think about it

Mental model

Imagine

A concert door policy

Think of

Rate limits & retries

The venue lets people in as others leave. Pushing harder doesn't speed entry — it starts a line fight. The right move is to queue politely (backoff) and leave if your ticket is invalid (4xx).

Mechanics

How it works

  • Each request counts at the provider; hitting the ceiling returns 429 with a Retry-After header.
  • The client increments backoff: delay = base * 2^attempt plus random jitter 0..delay/2.
  • Retryable classes: 429 (respect Retry-After), 408, 5xx. Non-retryable: 400, 401, 403, 422.
  • Batching and concurrency semaphores keep you under the ceiling rather than bouncing off it.
  • Distributed clients need coordinated limiting or per-instance budgets to avoid aggregate abuse.

A disciplined retry timeline.

Request 1
429 + Retry-After
Wait
Request 2
500
Succeed or give up
Hands-on

Examples

Backoff with jitter and Retry-After

Backoff with jitter and Retry-After
import time, random
 
def call_with_retry(st, attempts=5):
    for i in range(attempts):
        resp = st()
        if resp["status"] != 429 and resp["status"] < 500:
            return resp
        wait = float(resp.get("retry-after", random.uniform(0.5, 1.5) * (2 ** i)))
        time.sleep(wait)
    raise RuntimeError("max retries exceeded")

Honors Retry-After when present; otherwise exponential backoff with jitter. 4xx non-transient statuses return immediately instead of wasting calls.

Real world

In production

  • A backfill job paces embedding calls under the tokens/min ceiling, using a semaphore plus jittered retries so it never trips the limiter.
  • A gateway centralizes rate limiting per API key and queueing, protecting both the provider and the downstream service.
  • An agent loop that makes tool-driven model calls caps concurrent in-flight requests to keep under interactive limits.
Guidelines

Best practices

  • Read and honor rate-limit headers (x-ratelimit-remaining, Retry-After).
  • Cap attempts and surface the failure after the budget is spent.
  • Add jitter to every retry — synchronized retries create thundering herds.
  • Monitor your 429 rate: chronic 429s mean you're over-provisioned, not unlucky.
Watch out

Common mistakes

  • Retrying instantly, guaranteeing another 429.
  • Retrying 400/401/403 — they never succeed, and every attempt burns budget.
  • No max attempts, so a dead provider spins forever costing time and money.
  • Ignoring token-per-minute limits and only counting requests, then still getting throttled.
Root cause analysis

Common issues

Error
429s even at low request counts
Root cause

Token-per-minute limit exceeded by long prompts/contexts, or a shared pool exhausted by another service

Fix

Track estimated tokens per call, reduce concurrency, and check pool-sharing.

Error
Retry storm after a provider outage
Root cause

All clients retried at the same interval without jitter

Fix

Add jitter and stagger; consider a circuit breaker at the gateway.

Error
Pipeline stuck retrying a dead endpoint forever
Root cause

Infinite retries with no attempt cap

Fix

Cap attempts, surface the error to an alerting pipeline, and fail fast on 5xx after N tries.

When things break

Debugging tips

  • A sudden spate of 429s at a fixed clock time usually means your limits reset on that boundary.
  • Inspect Retry-After values across providers — they differ, and hardcoding 'wait 1s' is fragile.
  • Plot 429 rate vs concurrency to find your sustainable ceiling.
  • If failures come in waves, look for a shared token bucket at a gateway you forgot about.
Interviews

Interview checkpoint

Practice

Hands-on lab

Emulate a Provider Rate Limiter
30 min · intermediate
In progress
Objectives
  • Simulate a token-bucket limiter returning 429 + Retry-After
  • Smooth request pacing to stay under the ceiling
  • Compare naive vs jittered backoff on outage recovery

Emulate a Provider Rate Limiter

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which status is safely retryable?

2

Jitter in backoff prevents synchronized retry storms.

Go deeper

Official documentation

OpenAI rate limits guide
docs
AWS retry patterns (backoff/jitter)
docs