Production LLM Architecture
The request's journey, made robust
A production LLM stack threads request flow through gateway, context assembly, model calls, guardrails, tracing and storage. Stateless compute, redundant providers, prompt and model versioning, and evals gate every change.
What you will learn
- Trace a request through the full production stack
- Position the gateway in the request flow
- Chain guardrails, caching and evals around the model
What is this?
The architecture separates the app from the model behind a gateway: requests are classified, routed, cached and versioned; context is assembled (history, retrieval), the model call is made with fallbacks, output passes guardrails, and every step is traced with its version and cost.
Why do you need it?
- Direct app→model calls couple you to one vendor's API and pricing.
- Models change; versioning the prompt and config must mirror versioning the code.
- Outputs are untrusted until guardrails and evals say otherwise.
Mental model
LLM stack
A well-run kitchen
Orders enter at the pass (gateway), prep stations (context), the chef (model) fires with substitutes if the special runs out (fallbacks), and a final check (guardrails) catches anything wrong before it leaves.
How it works
- Gateway: auth, rate-limit, routing, caching, cost tagging, provider failover.
- Assembly: history windowing, retrieval calls, prompt templating per version.
- Model call: primary + fallback, streaming, timeouts, retries.
- Post-output: guardrails, scoring, storage of traces for evals and replays.
One request, end to end.
In production
- A chat product routes through a gateway that rate-limits per tenant and logs cost per version.
- An agent app assembles context from a vector store and tools, all traced.
- A summarizer fans through a fallback chain (primary provider → secondary → batch).
Best practices
- Keep compute stateless; the source of truth is the store and the versioned prompt.
- Version the prompt and model alongside deploys; traces carry both.
- Put safety rails before and after the model, not only after.
- Automate evals into the release path so architecture changes are gated.
Common mistakes
- Embedding provider credentials and prompts directly in the app.
- No fallback path — one provider's outage is your outage.
- Skipping guardrails because 'the model is safe by default'.
Debugging tips
- Everything fails after a deploy → compare trace version fields pre/post change.
- Prompts and responses differ from prod → replay the stored trace, not a fresh guess.
- One stage dominates latency → check its own spans and timeouts.
Interview checkpoint
Hands-on lab
- Model the request pipeline as stages
- Tag each stage with duration
- Spot the bottleneck numerically
Trace a Request Through the Stack
30 min · beginner
Quick quiz
What belongs between the app and the model provider?
Outputs should pass guardrails only after the model call.