Resilience and Fallbacks

• Tune timeouts and retries for streams
• Wire provider fallback with circuit breaking
• Degrade gracefully to non-model paths

• Provider regions and models can all 500 at once; a single chain is fragile.
• Hung streams without timeout produce silent user wait and cost leakage.
• JSON/garbage outputs are a reliability event just like an error code.

You plan a primary route, a marked detour (provider B), signs when to slow (timeouts), and finally a footpath into town (degraded answer) instead of stranding the traveler.

• Set connect/read timeouts sized above p99 latency; stream-aware, not total-only.
• Retry 429/5xx with jittered exponential backoff and a cap cheap enough to survive.
• Gateway fallback to a second provider or a retry with the repair step on bad output.
• Degrade path: cache hit → small model → deterministic rule/policy answer.

• A chat app fails over openai → anthropic → small cached answers on outage.
• A summarizer validates JSON and re-prompts once with the parse error included.
• A copilot returns 'here's the plan' from cached snippets when the model tier is 429ing.

• Make every fallback real: test it by killing the primary, not in theory.
• Use short retries and cap the total budget; a long retry storm is worse than one timeout.
• Return degraded-but-useful results rather than empty failures where possible.
• Log which fallback served each request — the trace tells you the health ladder worked.

• No timeouts → hung streams burn users and budget.
• Retry loops on 4xx-like prompt errors (they won't heal by retrying).
• Fallback to a weaker model with no quality gate — a 'successful' response that's worse than failure.

• Fallbacks never fire → circuit breaker always open; test with fault injection.
• Latency spikes look like provider issues → check gateway retry storms colliding.
• Degraded answers surprise users → surface the fallback as a 'lite' response in the UI/trace.