GenAI Gateway
One door for every model call
The gateway is the single front door for model traffic: unified auth, routing, provider failover, rate limiting, caching, cost logging and observability. It turns provider-account sprawl and secret-in-app code into one controlled point.
What you will learn
- Design the gateway's responsibilities
- Wire provider failover and retries
- Enforce auth, limits and cost tags centrally
What is this?
A GenAI gateway is a reverse proxy with model-specific intelligence: it holds provider keys and models, exposes one OpenAI-compatible API, routes by policy, fails over when a provider errors or rate-limits, caches, streams, logs cost, and enforces per-tenant limits. Apps never touch provider credentials.
Why do you need it?
- Direct SDK usage scatters secrets, pricing and fallback logic through the codebase.
- Provider incidents need a switch, not a code push.
- Rate-limiting and cost attribution belong at a central point.
- A single API shape lets one client reach many providers.
Mental model
Gateway
A company reception desk
Everyone gets screened, signed in, and routed to the right department by one receptionist — instead of wandering the halls asking departments for access.
How it works
- Clients talk to /v1/chat/completions with their own app key.
- Gateway authenticates, loads the chosen provider+model policy, and forwards.
- On 429/5xx/timeout it fails over to the next configured provider/model.
- Responses stream back; usage is tallied to the tenant's cost ledger.
Gateway fan-out.
In production
- A startup points every service at the gateway, swapping to a cheaper model with one config.
- A platform fails over openai → anthropic → self-hosted during pricing spikes.
- An enterprise enforces per-tenant token budgets and content rails in gate.
Best practices
- Expose one compatible API; keep per-provider knobs in config, not app code.
- Timeouts and short retries at the gateway; failover on 429/5xx.
- Tag every request with tenant, model, endpoint for the cost ledger.
- Stream to the client; buffer minimally so TTFT stays honest.
Common mistakes
- Rate-limiting at the app instead of the single door — bypasses and double counting.
- Failover that never exercises the backup, so it 500s when finally used.
- Gateway becoming a second app with its own unversioned prompt logic.
Debugging tips
- All calls fail → check the gateway's provider policy and key status first.
- Latency spikes through the gateway → check its buffering/streaming path, not the provider.
- Cost ledger off → confirm the gateway tallies usage from responses, not guesses.
Interview checkpoint
Hands-on lab
- Wire a primary and fallback provider
- Fail over on simulated 429 and 5xx
- Log which provider served each request
Failing Over Between Providers
35 min · intermediate
Quick quiz
A gateway's primary job in front of providers is?
Apps should hold provider API keys when a gateway exists.