LLM model routing and fallback

Filter by eligibility before preference

A router may consider task type, cost, latency or capability, but those preferences belong after hard constraints. Which providers may receive this data class? Which tools and output schemas are required? Is the model tested for this task? A route that violates policy is not an inexpensive alternative.

AWS describes several multi-model routing strategies, including approaches that select a model based on task characteristics. The application still needs its own eligibility and evaluation contract; a routing pattern does not establish that a particular provider is appropriate for the data or task.[1]

Version the route as part of the product

A route is more than a model name. It includes the prompt, tool definitions, output contract, decoding settings where applicable, limits and policy version. Different providers can disagree about structured output, tool-call handling and error behavior. Normalize the application contract, then test each adapter against it.

task_class
data_policy_class
eligible_routes[]
selected_route + version
selection_reason
fallback_policy_version
terminal_outcome

Record enough to explain a route choice without storing sensitive input in the log. If the product changes models automatically, the release evidence should make that behavior visible. A buyer should not discover during an incident that the fallback was never evaluated.

Worked scenario: a blocked fallback

A synthetic document-analysis task is allowed on route A under the product’s data policy. Route B is cheaper and performs well on public test documents, but is not approved to receive this document class. If A is unavailable, B is ineligible. The system should queue within its stated budget, use another eligible route or return an explicit unavailable state.

Now consider route C, which is eligible but has not been tested with the task’s required structured tool result. It should not become the automatic fallback solely because it accepts text. First verify the complete task, including malformed outputs, refusal behavior and failure handling.

Classify failure before changing routes

A model timeout before any tool action is different from a timeout after a business mutation may have committed. Retrying the latter through a new model can produce a second action. Preserve the intended operation ID and reconcile it through the domain service before continuing.

Idempotent API design ties repeated requests to the same intended operation and requires appropriate duplicate handling. That concern remains even when the new attempt is initiated by a different model or provider.[2]

• Transient model failure before a side effect: retry or use an eligible tested fallback within the task budget.
• Invalid structured result: record the failure and follow a bounded repair policy; do not create an unlimited model loop.
• Policy refusal or denied tool action: respect the boundary; changing providers is not a way around it.
• Unknown tool outcome: reconcile the original operation before any new mutation.
• Task budget exhausted: return an unresolved state with the evidence already collected.

Evaluate routes by task slice

A global average can hide a route that works well for simple summaries and poorly for exception handling. Compare routes on the task classes that matter to the product. Include long inputs, ambiguous cases, tool failures and the data formats users actually supply.

Measure accepted outcomes, review effort, useful completion time and cost per accepted task. A cheaper first attempt that requires more repair may be more expensive overall. Keep the task set and rubric comparable, and record model and configuration changes.

When one route is the better choice

Use one well-tested route when traffic is modest, task variety is narrow or routing adds more maintenance than value. Introduce another route for a specific measured benefit or resilience requirement. The goal is a product that behaves predictably across its permitted paths, not the largest possible provider menu.