Model Routing and Tiers
Don't ship every request first-class
Route easy requests to cheap small models and hard ones to frontier models. Classify by intent, size or confidence; escalate on failure. Tiers create 5-20x price spreads for work that mostly doesn't need the big model.
What you will learn
- Design a routing heuristic for request tiers
- Compose fast-cheap and premium paths
- Add a fallback escalation on failure
What is this?
Model routing picks a model per request based on cheap signals — intent, request length, language, detectability of the task. A cascade upgrades when a small model's output fails confidence checks. The win: most traffic runs on the cheap tier while quality-critical requests pay for the premium one.
Why do you need it?
- Small fast models handle the long tail of easy requests at a fraction of the price.
- Hard examples are rare; paying frontier prices for all of them is waste.
- Escalation gives a safety net for the cases the cheap model gets wrong.
Mental model
Model routing
Hospital triage
A nurse (cheap signal) sends flu cases to the clinic and chest pain to the ER — resources match urgency instead of everyone seeing the specialist.
How it works
- Classify each request: intent, complexity, required skill.
- Map classes to tiers: tiny model, mid model, frontier.
- Route; on low-confidence output, escalate to the next tier.
- Instrument cost and quality per tier to tune thresholds.
Tiered path.
In production
- A chat app sends greetings and FAQ to a 10M-parameter model, deep technical to frontier.
- An extraction service runs a fast model and re-runs failures with the big model.
- A translation platform lets users pick speed vs quality tiers at different prices.
Best practices
- Route on cheap, fast signals — parsing, length, keyword, cache membership.
- Escalate with the failed output as context so the premium model can correct it.
- Adjacent fallbacks (retry on another provider) keep single-vendor failures from stalling.
- Monitor each tier's cost and quality; drift your thresholds with data.
Common mistakes
- Complex AI to route the router (pays for the classification more than the routing saves).
- One-way routing with no escalation — a misroute is a dead end.
- Routing on intent alone while ignoring request size and required skill.
Debugging tips
- Cheap tier quality drops → threshold too low; bring borderline traffic up a tier.
- Escalation rates through the roof → the classifier or threshold is misfiring; inspect samples.
- Late prompts arriving at the big model → add a size-based early tier.
Interview checkpoint
Hands-on lab
- Classify requests into easy and hard
- Route each to its tier
- Escalate a flagged failure to the premium model
A Two-Tier Router with Fallback
35 min · intermediate
Quick quiz
What do you need in a routing system besides a classifier?
Routing should always send hard requests straight to the biggest model.