Model Routing and Fallbacks
On this page
A company sends every request from its assistant to the largest model it can buy. The monthly bill is growing faster than usage, though most questions are simple ("what are your opening hours?"). Then the provider has a two-hour regional outage and the assistant is simply down, because there is no plan B. Both problems have the same fix: put a routing layer between the application and the models, so each request goes to the cheapest model that can handle it, and failures fall through to another model, provider or safe response.
How it works

Where the router lives
Usually inside an LLM gateway: the application calls one internal API (through the API Gateway), and the gateway handles authentication, rate limits and quotas, logging and tracing, caching, routing and fallbacks. Applications never hard-code a provider.
Routing: which model for this request?
- Static rules: by feature, tenant plan, region (data residency), language, or input length. Simple, predictable, and often enough.
- Classifier routing: a small, fast model or classifier predicts difficulty or intent and picks the model tier. Train it on requests labeled by which model answered acceptably (from evals or judges).
- Cascade: try the small model first; check the answer (a verifier, a confidence score, a judge, schema validation, tests for code); escalate to the large model only when the check fails. Pays twice on escalated requests.
- Constraints: budget per tenant, latency SLOs (a voice feature can't wait for the biggest model), and current health of each model.

In the included routing_sim.py (example prices and quality numbers, 25% of requests "hard"): always using the large model costs $6.00 per 1,000 requests at 94.5% quality; always small costs $0.40 at 84.3%; a 90%-accurate router costs $2.08 at 92.3%; a small-then-large cascade costs $1.89 at 91.7%. About two thirds of the cost goes away for about 2 points of quality, and a better router narrows the gap. Measure your own trade-off on your eval set: the numbers depend entirely on your traffic.
Fallbacks: what happens when a call fails?

- Primary model.
- Retry once with jittered backoff, only for retryable errors (rate limit 429, server errors, timeouts).
- Secondary: the same model in another region, or a comparable model from another provider.
- Degrade: a smaller model or a shorter answer, clearly acceptable for this feature.
- Safe failure: a cached answer, a "please try again" message, or handoff to a human.
Supporting mechanics:
- Timeouts on time-to-first-token, not only total time, since a stream can stall.
- Circuit breakers: after repeated failures, skip the unhealthy provider for a cooling-off period instead of making every request wait for a timeout.
- Idempotency and streaming: if a stream fails halfway, decide whether to restart on the fallback (and discard partial output) or show what you have.
- Don't fall back on non-outages: a safety refusal, invalid input or context-length error won't be fixed by another provider and may bypass your safety policy.
With independent failures, a provider at 99.5% availability plus one fallback gives about 99.997%. In practice failures correlate (shared regions, shared traffic spikes), so test fallbacks for real.
Trade-offs and costs
- Quality drift: prompts tuned for one model behave differently on another. Each fallback model needs its own prompt variant and evals (see Versioning Prompts and Models).
- Router errors send hard requests to weak models; monitor quality by route, not just overall.
- Cascade latency: escalated requests pay for two model calls.
- Complexity and consistency: users may notice different styles from different models; keep output formats and tone aligned.
- Data and contracts: a fallback provider must meet the same privacy, residency and compliance terms as the primary.
In production
- Log the chosen route, model, provider and reason on every trace (see LLM Observability and Tracing).
- Dashboards by route: quality (judge scores, feedback), cost, latency, fallback rate, circuit-breaker state.
- Run fallbacks regularly (a small share of traffic, or game days), so they aren't broken when you need them.
- Re-train or re-tune the router as models and traffic change; re-run evals on every model in the chain.
- Per-tenant budgets: route to cheaper tiers or slow down when a tenant nears its limit.
When to reach for it
Any product with meaningful LLM spend, mixed-difficulty traffic, or availability requirements beyond a single provider's SLA. Also central to "design an LLM gateway" and "cut LLM cost 10×" interview questions.
Example
The company from the opening adds a gateway. Rules first: FAQ and order-status intents go to a self-hosted small model with retrieval; a small classifier sends the rest to the small or large model; code questions always go to the large model. Fallbacks: large model in region 2, then another provider's comparable model with its own tested prompt, then a "we're having trouble" message with a link to help pages. Circuit breakers trip after 20 failures in 30 seconds. Results: spend down 58%, judge-scored quality down 0.8 points, and the next provider outage shows up only as a fallback-rate spike on the dashboard. (Example numbers.)
Common mistakes
- Hard-coding one provider in application code.
- Retrying forever or retrying non-retryable errors.
- Untested fallbacks, or fallback models running prompts written for a different model.
- Falling back on safety refusals, effectively shopping for a model that will comply.
- Measuring only average quality, hiding a weak route.
- No timeouts on the first token, so a hung stream blocks the fallback.
Say this in the interview
"I'd put routing in an LLM gateway behind the API gateway. Routing uses rules first (feature, tenant, region), then a small classifier or a cascade that tries a cheap model and escalates when a verifier or judge rejects the answer, within budget and latency limits. Fallbacks go primary, one jittered retry for retryable errors, a secondary region or provider with its own tested prompt, a smaller model, then a safe failure, with first-token timeouts and circuit breakers. Every trace records the route, and I'd track quality, cost and fallback rate per route and exercise the fallbacks regularly."
Interview questions on this
- Router vs cascade: when would you use each? Router when difficulty is predictable from the input; cascade when you can cheaply verify outputs.
- How do you train a routing classifier? Label requests by the cheapest model whose answer passes evals or a judge; train on features of the request.
- Which errors should trigger a fallback? Timeouts, 429s, 5xx, provider outages; not refusals, invalid input or context-length errors.
- What's a circuit breaker for? Stop sending traffic to a failing provider so requests don't each wait for a timeout.
- How do you keep quality consistent across fallback models? Per-model prompt variants, evals for each, aligned output schemas.
- How would you route under a per-tenant budget? Track spend in the gateway, shift to cheaper tiers or queue as the tenant nears its limit.
Used in
- AI Engineering: How Would You Cut LLM Cost by 10x?; Choosing a Model; Token Rate Limits and Quotas; Semantic Caching for LLM Apps; LLM Observability and Tracing; Versioning Prompts and Models, and Rolling Them Out.
- AI System Design: Design an LLM Gateway; Design an Inference Platform; Design ChatGPT.