Fallback Chains

What a fallback chain is

A fallback chain is the ordered list of model providers a request may be sent to. The gateway sends the request to the first provider in the chain. If that provider fails in a way that is worth another try, the gateway sends the same request to the next one, up to the key’s attempt limit. The chain for a request is built from three things:

The key’s routing setting: no fallback, fall back to all eligible providers, or a routing policy.
The requested model: only providers that can serve the model stay in the chain. A request for claude-sonnet-4-5 is never sent to an OpenAI credential.
The key’s provider access and budgets: a provider the key is not allowed to use, or whose provider-scoped budget is exhausted, is removed.

Routing settings on a key

Open the key under AI Gateway, Virtual Keys and click Edit. The Routing section lists two fixed choices, then one row per routing policy the key can reach:

Choice	Chain
Default: no fallback	One attempt. The provider’s error reaches the caller unchanged.
Fall back to all eligible providers	Every provider the key’s scope can reach, ordered by each provider’s Fallback priority (Settings, Model Providers, Advanced), lowest first, then by creation date.
A routing policy	The providers the policy lists, in the policy’s order. See Routing policies.

The attempt limit is 3 by default and is part of the key’s configuration (fallback.maxAttempts on the management API). With no fallback the limit is 1.

What triggers a fallback

The gateway decides from the provider’s real answer, and this is not configurable.

Provider outcome	Next provider?
HTTP 5xx	Yes
HTTP 429	Yes
HTTP 404	Yes. In a multi-provider chain a 404 usually means this provider does not serve that model.
Timeout, connection failure	Yes
Any other 4xx (400, 401, 402, 403)	No. The error reaches the caller.
Credential or provider configuration refused before the call	No. Retrying gives the same answer.
The caller disconnected	No

When every attempt fails, the caller gets the error of the last provider tried, with that provider’s status and body.

Streaming

For a streaming request the gateway falls back only while opening the stream. Once the first bytes reach the caller, a provider failure ends the stream with an event: error frame. Chunks from two providers are never spliced into one response.

What the caller sees

The response carries X-LangWatch-Fallback-Count with the number of failovers that happened, and only when it is at least 1. A request served by the first provider has no header.

HTTP/1.1 200 OK
X-LangWatch-Fallback-Count: 1
X-LangWatch-Gateway-Request-Id: req_3e8385c4e62cbe4e31ca9c410c5891

The body is the provider’s normal response and does not name the skipped provider. The span records the provider that served the request. Prometheus counts every attempt in gateway_provider_attempts_total{credential_id, outcome} and every failover in gateway_fallback_events_total{from_credential, to_credential}.

Circuit breaker

Each provider credential has a circuit breaker. After 10 failures inside a 30 second window the breaker opens and the gateway skips that credential for 60 seconds, then lets one probe request through. A 5xx, a timeout or a connection failure counts as a failure; a 429 or a terminal 4xx does not. The window decides: the same ten failures spread over more than 30 seconds leave the breaker closed. A skipped credential does not consume an attempt. When the breaker is open on every credential in the chain the caller gets 503 circuit_open with the message provider temporarily unavailable: repeated upstream failures opened the circuit breaker, retry shortly. The window, the threshold and the open duration are gateway settings: LW_GATEWAY_CIRCUIT_WINDOW_S, LW_GATEWAY_CIRCUIT_THRESHOLD, LW_GATEWAY_CIRCUIT_COOLDOWN_S. gateway_circuit_state{credential_id} reports 0 closed, 1 open, 2 half-open, and is the only view of the breaker state. A skipped attempt is counted as outcome="circuit_open" in gateway_provider_attempts_total.

Errors before the first attempt

Code	Status	Cause
`model_not_allowed`	400	The model resolves to a provider the routing policy does not list. The message names the provider and the policy.
`policy_violation`	403	A restriction on the policy refuses the model, or the model is not in the policy’s allow-only list. The message names the model: `model "gemini-2.5-flash" is blocked by policy`, or `model "gemini-2.5-flash" is not in allowlist`. See Policy rules.
`model_not_allowed`	400	The model is outside the key’s own model allowlist, or resolves to a provider the key’s provider access excludes.
`no_provider_configured`	400	The key reaches no provider at all.
`model_not_recognized`	400	No provider on the key declares the model. See Model naming.

Boundaries

The gateway sends each attempt to one credential. Requests are not spread across several credentials of the same provider.
Which failures trigger a fallback is fixed. A per-key trigger list does not exist.
A fallback to a different vendor sends the prompt to that vendor. Use a routing policy or provider access to keep a key inside one vendor.

Also check: Routing policies, Model naming, Troubleshooting.

Get Started

SDK Integration

Virtual Keys & Budgets

Billing & Metering

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

What a fallback chain is

Routing settings on a key

What triggers a fallback

Streaming

What the caller sees

Circuit breaker

Errors before the first attempt

Boundaries

​What a fallback chain is

​Routing settings on a key

​What triggers a fallback

​Streaming

​What the caller sees

​Circuit breaker

​Errors before the first attempt

​Boundaries

What a fallback chain is

Routing settings on a key

What triggers a fallback

Streaming

What the caller sees

Circuit breaker

Errors before the first attempt

Boundaries