Skip to main content
A fallback chain is an ordered list of provider credentials the gateway tries in sequence when the primary fails. It’s per-virtual-key configuration; each VK owns its own chain.

​
Configuring a chain

On the VK edit screen:
  1. Select a primary provider credential.
  2. Add one or more fallback credentials in the order they should be tried.
  3. Pick which conditions trigger fallback (by default: 5xx, timeout, rate_limit).
  4. Set timeout_ms (default 30000) and max_attempts (default 3).

​
When fallback triggers

​
Model translation across providers

A single VK may have mixed providers (OpenAI + Anthropic + Bedrock). The gateway uses Bifrost’s provider-dispatch library to translate payloads: same messages schema, different wire formats. If the client requests gpt-5-mini and fails over to Anthropic, the gateway applies the VK’s model_aliases to pick the Anthropic equivalent (e.g. claude-haiku-4-5-20251001). Configure this per VK:
The :fallback suffix is optional, if absent the gateway uses the same model name against the next provider and expects it to exist.

​
Streaming

Fallback behaviour differs based on when failure occurs:
  • Before first chunk emits → transparent fallback. The stream-setup call (bifrost.ChatCompletionStreamRequest) walks the chain the same way non-streaming dispatch does; the client sees a single stream from whichever slot accepted the request. X-LangWatch-Fallback-Count reports the skipped slot count.
  • After first chunk has streamed → no mid-stream fallback. The gateway emits a terminal event: error frame (with code: upstream_mid_stream_failure) and closes the connection. The client may retry; a fresh request would then re-walk the chain.
This split is deliberate, splicing chunks from two providers would produce an inconsistent response with mismatched tool-call ids and accumulated-content replay. It’s a hard “no” per contract and enforced by a byte-exact assertion on the SSE error frame shape so future refactors can’t accidentally break it. See Streaming → pre-connection fallback, mid-stream failure for the exact frame bytes and a worked example.

​
Observing the chain in traces

Fallback attribution today:
  • Prometheus counter gateway_provider_attempts_total{credential_id, outcome} increments once per attempt, with outcome in success | fallback_success | retryable_5xx | not_found | rate_limit | timeout | network | circuit_open | non_retryable | chain_exhausted | context_done.
  • Response header X-LangWatch-Fallback-Count: N, how many fallbacks were attempted before success.
  • Request-id correlation via X-LangWatch-Request-Id, join the metric + log line back to the specific trace.
Per-attempt nested spans (langwatch.fallback.attempt, .reason attrs) are a v1.1 observability follow-up. In v1, the counter + header give you the aggregate picture; per-attempt reasoning lives in the gateway log line for that request-id.
If all attempts in the chain fail, the gateway returns the last provider’s error envelope mapped to the OpenAI-compatible shape (provider_error or upstream_timeout as the type).

​
Circuit breaker

Each provider has an independent circuit breaker with a sliding window: Override at the service level: Per-replica, not shared: each gateway replica maintains its own breaker state. This is deliberate, under a large-scale outage, N replicas rediscovering the recovered provider independently is resilient; depending on Redis for breaker consensus is not. Circuit state is emitted as a Prometheus gauge gateway_circuit_state{credential_id}, one series per credential slot. The state is encoded in the gauge value, not in a label: 0 closed, 1 open, 2 half-open. Alert on gateway_circuit_state == 1 for 5m to catch real provider outages (as distinct from transient blips). When a request hits a circuit that’s currently open, the gateway skips to the next entry in the fallback chain immediately, no wasted round-trip to a provider we already know is down.

​
Sizing the chain

Diminishing returns after 3 entries. With the default timeout_ms=30000 and max_attempts=3, the worst-case wall-clock to exhaust the chain is ~90s; the latency budget of the original call is gone well before then and the client has probably given up. Use chains of 2-3 for latency-sensitive traffic and lower per-entry timeout_ms if you need a tighter total budget; longer chains are fine for batch/offline.