Model Serving
Intermediate 12 min read Model Serving

Serving Gateways

One contract, many upstreams

A gateway fronts your model calls with routing, retries, rate limits, keys and observability so clients depend on one contract regardless of upstream. It is the control plane between apps and the model fleet.

gatewayroutingAPI keyfailovermodel router
Prerequisites: Providers vs Self-Hosting
Before we start

What you will learn

  • Explain what a model gateway owns
  • Route by model id, cost and region
  • Add retry and failover without client changes
Introduction

What is this?

A model gateway is a reverse proxy for LLM calls: clients call one stable endpoint, and the gateway resolves model ids to upstreams (providers, self-hosted replicas), applies auth, rate limits, budget checks, retries and fallbacks, then streams back.

Motivation

Why do you need it?

  • Isolates clients from provider churn: swap Anthropic→OpenAI behind the same path.
  • Centralizes keys, budgets and per-team metering in one place.
  • Enables global policy — model allowlists, PII filters, cost caps — applied to every call.
Think about it

Mental model

Imagine

Gateway

Think of

An airline's booking system

Customers see one menu and one booking flow; behind it, the system picks actual flights, handles cancellations and rebooks across carriers.

Mechanics

How it works

  • Clients POST to /v1/chat with a model id the gateway recognizes.
  • The gateway checks the caller's key, rate limit and budget.
  • It resolves the model to one or more upstreams, with priority and health.
  • It can retry on timeout/429/5xx, stream through, and log usage.

Request path through the gateway.

App
Gateway
Failover
Upstreams
Hands-on

Examples

Routing by model id and region

Routing by model id and region
routes:
  "fast-model":
    - upstream: "selfhost-eu"        # primary for EU traffic
      weight: 80
    - upstream: "openai-fallback"     # provider spare
      weight: 20
  "frontier-model":
    - upstream: "anthropic-1"
      weight: 100
retry: { strategy: "exponential", max_attempts: 3, on: [429, 502, 503] }

Model ids stay fixed for clients; weights, regions and fallbacks all live in gateway config.

Real world

In production

  • An org runs all internal apps through one gateway with per-team keys and quotas.
  • Global traffic routes to regional upstreams to bound latency and data-residency.
  • A cost guard rails off expensive models per tenant and triggers alerts at 80% budget.
Guidelines

Best practices

  • Keep the model id in the client stable; change behavior in gateway config.
  • Expose stream-through so failover preserves the SSE contract.
  • Log a canonical request id on every hop for triage.
  • Apply rate limits per key AND per gateway replica (use a shared counter).
Watch out

Common mistakes

  • Clients hardcoding upstream URLs, bypassing the gateway and its policies.
  • Retrying idempotently-safe GETs but also retrying prompts blindly on non-idempotent paths.
  • No budget enforcement, so a leaked key burns a large bill before revoke.
When things break

Debugging tips

  • If one upstream is slow, gateway p95 hides it — break latency down per upstream id.
  • Check the gateway's auth layer before blaming the model when a key is 'invalid'.
  • Stream-through breaks if the proxy buffers; confirm flush/stream config.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which problem does a gateway most directly solve?

2

Clients should call provider URLs directly for best flexibility.

Go deeper

Official documentation

LiteLLM proxy
docs
OpenAI gateway patterns
docs