System Design
Advanced 16 min read System Design

API Design at Scale

An API that works at small scale collapses at large scale through chatty endpoints, offset pagination on huge tables, missing rate limits and non-idempotent mutations. Cursor pagination, token-bucket rate limiting, idempotency keys and explicit versioning are the building blocks that keep a public API stable, fair and upgradeable under load.

RESTcursor paginationrate limitingtoken bucketidempotencyversioningopenapi429retrygRPC
Prerequisites: HTTP & the Web Load Balancing Caching Strategies
Introduction

What is this?

API design at scale is about shaping the contract so it survives heavy, adversarial traffic. Cursor pagination replaces OFFSET jumps that get slower as tables grow; rate limiting with a token bucket keeps any one client from consuming the whole platform; idempotency keys make retries safe for mutations; and explicit versioning lets the API evolve without breaking consumers. The same discipline applies to the wire contract (OpenAPI for REST, .proto for gRPC) so clients and servers can change independently.

Motivation

Why do you need it?

  • Deep OFFSET pagination scans and discards rows, degrading quadratically — a page 10,000 rows deep is catastrophically slow.
  • Without rate limits, one runaway or abusive client can consume every resource and starve everyone else.
  • Clients retry on timeouts; without idempotency, a retried payment creates a duplicate charge.
  • Versioned contracts let you ship breaking changes while old clients keep working on v1.
  • Well-documented contracts (OpenAPI/proto) let codegen generate clients, keeping consumers in sync automatically.
Think about it

Mental model

Imagine

A busy station with a single ticket window

Think of

Numbered carriages, queue gates and a receipt system

Cursor pagination is like numbered carriages — 'board carriage 14 next' tells you exactly where you are without recounting everyone in front. Rate limiting is the gate that keeps boarding orderly, and idempotency is a receipt system so a passenger who re-presents a paid receipt is not charged twice.

Mechanics

How it works

  • The API defines a resource model and verbs; every endpoint declares its request and response contract.
  • List endpoints paginate with an opaque cursor: the server returns a page plus a next_cursor token encoding position.
  • A gateway applies a token bucket per client key (API key or IP), rejecting excess requests with 429 and Retry-After.
  • Mutations require an Idempotency-Key header; the server caches the response for that key and replays it on retries.
  • Versions are explicit in the URL or headers, and breaking changes land in a new version with a deprecation deadline.
  • Both sides add resilience: clients use timeouts, retries with backoff and circuit breakers; servers shed load gracefully.

A request passes the rate limit, is served a cursor page, and a mutation is deduplicated by idempotency key.

Client
API gateway
Token bucket
429 + Retry-After
Service + data
Response
Reference

Key commands

curl 'http://localhost:3000/api/v1/orders?limit=20&cursor=1000'

Request the first page of orders after cursor 1000.

{"data":[...],"next_cursor":1020,"has_more":true}
curl -i -X POST http://localhost:3000/api/v1/orders -H 'Content-Type: application/json' -H 'Idempotency-Key: c7f8' -d '{"sku":"A1"}'

Create an order with an idempotency key; retrying this exact request returns the same result.

HTTP/1.1 201 Created
curl -i http://localhost:3000/api/v1/orders -H 'Authorization: Bearer tok'

Read the RateLimit-* response headers to see your remaining quota.

RateLimit-Limit: 100\nRateLimit-Remaining: 99
ab -n 5000 -c 50 http://localhost:3000/api/v1/orders

Flood the endpoint to trigger the 429 rate limit response.

Non-2xx responses: 4900
curl -i http://localhost:3000/api/v1/orders

Confirm the 429 response includes a Retry-After header.

HTTP/1.1 429 Too Many Requests\nRetry-After: 60
npx @redocly/cli lint openapi.yaml

Lint the OpenAPI contract to catch breaking or invalid schema changes in CI.

Your API description is valid.
Hands-on

Examples

Cursor pagination, rate limiting and idempotent creation in Express

Cursor pagination, rate limiting and idempotent creation in Express
// server.js - pagination, rate limiting, idempotent checkout
const express = require("express");
const rateLimit = require("express-rate-limit");
 
const app = express();
app.use(express.json());
 
// Token bucket style rate limit per client
const limiter = rateLimit({ windowMs: 60_000, max: 100 });
app.use("/api/", limiter);
 
// Cursor pagination over an ordered key (id)
app.get("/api/v1/orders", async (req, res) => {
  const limit = Math.min(Number(req.query.limit) || 20, 100);
  const cursor = Number(req.query.cursor) || 0;
  const rows = await db.orders.after(cursor, limit); // WHERE id > $cursor ORDER BY id LIMIT $limit
  res.json({
    data: rows,
    next_cursor: rows.length ? rows[rows.length - 1].id : null,
    has_more: rows.length === limit,
  });
});
 
// Idempotent creation: retries replay the original response
const seen = new Map();
app.post("/api/v1/orders", async (req, res) => {
  const key = req.headers["idempotency-key"];
  if (key && seen.has(key)) return res.json(seen.get(key));
  const order = await db.orders.create(req.body);
  if (key) seen.set(key, order);
  res.status(201).json(order);
});

The GET route clamps the page size and walks the table with WHERE id > cursor instead of OFFSET, so pages cost the same no matter how deep you go; next_cursor plus has_more tells the client exactly how to continue. The POST route requires an Idempotency-Key and caches the created resource, so a client that retries a timed-out request gets the original order instead of creating a duplicate. The rate limiter protects both routes by throttling each client to 100 requests per minute.

Real world

In production

  • Public APIs like Stripe and GitHub use cursor pagination and per-key rate limits so one abusive client cannot exhaust the platform.
  • Payment APIs require an Idempotency-Key on every mutation, and the server stores the response for 24h so retries are safe.
  • Internal services communicate over gRPC with HTTP/2 multiplexing and typed .proto contracts, while the public surface stays REST/OpenAPI.
  • Gateways (Kong, Envoy, AWS API Gateway) enforce rate limits, quotas and 429 handling without the service code knowing about them.
Guidelines

Best practices

  • Return 429 with a Retry-After header so clients know when to try again, and honor it on the client side.
  • Keep cursors opaque and stable; never let clients compute or skip to arbitrary positions with internal ids.
  • Require an Idempotency-Key for non-idempotent mutations and cache the response per key with an expiry.
  • Version the contract explicitly (v1/v2), announce deprecation with a deadline, and never silently change v1 semantics.
  • Set a hard page-size maximum, always return next_cursor and has_more, and document the pagination contract in OpenAPI.
Watch out

Common mistakes

  • Using offset/limit on large tables, so each deep page gets slower until the query times out.
  • No idempotency keys, producing duplicate charges or duplicate resources whenever a client retries.
  • Versioning only the URL but silently breaking the response shape, so clients break at runtime instead of at contract time.
  • Chatty fine-grained endpoints that force N+1 client calls, multiplying latency and backend load.
Root cause analysis

Common issues

Error
429s for a low-traffic client
Root cause

The rate limit is keyed too coarsely (all clients share one IP), or the window/burst is too small.

Fix

Key by API key rather than IP where possible, and tune the burst/refill to the legitimate client's peak.

Error
Duplicate order created on client retry
Root cause

The mutation is not idempotent and the client retried after a timeout.

Fix

Implement Idempotency-Key with a server-side response cache keyed by the header, persisted for the retry window.

Error
A list call returns tens of thousands of rows and times out
Root cause

The client ignored limit, or the server never enforced a maximum page size.

Fix

Clamp the page size server-side, reject requests above the max with 400, and push clients onto cursor pagination.

When things break

Debugging tips

  • Inspect RateLimit-* and Retry-After headers to confirm the client is being throttled and when it can retry.
  • Replay a mutation with the same Idempotency-Key and compare the response body — duplicates should be impossible.
  • Break 429 vs 5xx rates down by API key to find the client hammering the platform.
  • Walk a full result set with curl, following next_cursor until it is null, to verify the pagination loop terminates.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Scaled API Contract
45 min · advanced
In progress
Objectives
  • Implement cursor pagination over a growing data set
  • Add token-bucket style rate limiting with 429 + Retry-After
  • Make a mutation idempotent with an Idempotency-Key
  • Verify behavior under load with curl and a load generator

Build a Scaled API Contract

45 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Why is cursor pagination preferred over OFFSET pagination on large tables?

2

An Idempotency-Key header guarantees a mutation is executed exactly once even across concurrent retries.

3

A client times out on a payment call and retries the exact same request. Without any protection, what is the most likely outcome?

Quick reference

Cheat sheet

HTTP contract headers
Idempotency-KeyDeduplicate mutations on retry
Retry-AfterSeconds until the client may retry
RateLimit-LimitQuota for the window
RateLimit-RemainingTokens left in the window
Cache-ControlWhat the edge may cache
VaryCache-key-relevant headers
Pagination contract
?limit=20Page size, clamped server-side
?cursor=1000Opaque position token
next_cursorContinue from this value or null
has_moreWhether more pages exist
WHERE id > cursorIndex-friendly page query
Go deeper

Official documentation

OpenAPI Specification
spec
gRPC — Introduction
docs
Stripe — Idempotent Requests
docs
Stripe — Rate Limits
docs