Serving Tier Design

• Design request routing and feature fetch for an online model
• Tune dynamic batching for throughput under latency budgets
• Build health checks, fallbacks and graceful degradation

• A model with great quality is worthless if it can't meet the product's latency budget.
• Feature fetching is usually the hidden cost — network round-trips to feature stores dominate request time.
• Tail latency (p99+) is what users actually feel and what autoscalers react to.
• Graceful degradation decides the revenue impact of an outage: cache last-known-good, fallback model, or a fixed rule.

Orders (requests) arrive in bursts, each needs ingredients (features) assembled quickly, and the whole kitchen must keep serving even when one station breaks. The head chef is your latency budget.

• A request hits the API gateway, which routes by model/version and enforces authN/authZ and rate limits.
• The serving layer fetches features: from the online feature store by key, from request payload, or cached — minimizing round-trips.
• The model server executes inference, optionally coalescing multiple requests into one batch for GPU/TPU efficiency.
• The response is returned with latency/quality telemetry; a circuit breaker and fallback path catch slow or failing models.
• Health checks (liveness/readiness) drive load balancers and autoscalers; a cold/unready pod never receives traffic.

• A search ranking model batches up to 64 requests into one Triton call, cutting GPU cost per query by 70% within the 250ms budget.
• A recommendation API reads features from Redis, with a payload-included feature fallback so a feature-store outage degrades quality instead of availability.
• A checkout fraud scorer runs a cheap rules model as a circuit-breaking fallback so a model server blip never blocks a transaction decision outright.

• Design to a latency budget with headroom: if the app allows 300ms, budget the model at 150ms.
• Fetch features in parallel and cache aggressively; each network hop is a latency tax.
• Use dynamic batching with a max-batch-wait — never batch at the expense of the latency budget.
• Give the serving tier a degradation ladder: live model → cached prediction → fallback model → safe default.
• Expose readiness based on model load, not just process start.

• Tuning for mean latency while p99 blows past the budget during traffic spikes.
• A feature fetch path that doubles request time because features are fetched serially.
• No graceful degradation, so an inference outage converts an availability problem into a correctness black-out.
• A readiness probe that says ready before the model weights are loaded, so pods serve 5s errors.

• Profile the request with distributed tracing: feature fetch time vs model time vs serialization.
• Watch p99 separately from mean; gradient traffic will expose the tail.
• Check batching wait time — requests waiting for a batch that never fills waste the entire latency budget.
• Load-test with a production-like feature latency, not a mock that returns in 0ms.