> ## Documentation Index > Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Fallback Chains > Survive provider outages, rate limits, and timeouts transparently. A fallback chain is an ordered list of provider credentials the gateway tries in sequence when the primary fails. It's per-virtual-key configuration; each VK owns its own chain. ## Configuring a chain On the VK edit screen: 1. Select a **primary** provider credential. 2. Add one or more **fallback** credentials in the order they should be tried. The VK's resolved configuration carries the chain and the attempt limit: ```json theme={null} { "fallback": { "chain": ["pc_primary_openai", "pc_anthropic", "pc_bedrock_us_east"], "max_attempts": 3 } } ``` `chain` is the ordered credential list. `max_attempts` (default `3`) sets how many of them the gateway tries for one request. A key with no fallback gets exactly one attempt. ## When fallback triggers The gateway decides which failures move to the next provider. It reads the real upstream result, and the rule is the same for every virtual key. There is no per-key trigger list. A narrowed list could only turn a failure the gateway can recover from into one that your callers see. | Condition | Fires? | Why | | ------------------------------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Upstream `5xx` | ✅ | Provider's fault; next provider may work. | | `timeout` | ✅ | Provider degraded. | | `429 rate_limit_exceeded` | ✅ | Primary is throttled; secondary may have headroom. | | `network_error` (DNS/TCP/TLS) | ✅ | Connectivity issue to primary. | | Upstream `404 Not Found` | ✅ | In a chain with more than one provider, this usually means that this provider does not serve that model. The next one can. Common with self-hosted and OpenAI-compatible providers. | | `circuit_breaker` open | ✅ | Preemptive, gateway knows primary has been failing recently. | | Upstream `400 Bad Request` | ❌ | Client-fault. Surfacing the error is correct. | | Upstream `401 Unauthorized` | ❌ | Provider credential bad. Needs human fix, not masking. | | Upstream `403 Forbidden` | ❌ | Authorization issue. Silent switch would hide a real problem. | | Client disconnect or an expired request deadline | ❌ | The caller went away. This tells nothing about the provider's health. | | LangWatch-internal `invalid_api_key` etc | ❌ | Never reaches the fallback layer. | ## Crossing providers mid-chain A single VK may have mixed providers (OpenAI + Anthropic + Bedrock). The gateway uses [Bifrost's provider-dispatch library](https://github.com/maximhq/bifrost) to translate payloads: same `messages` schema, different wire formats. The model does not change across the chain. One request resolves to one model. The gateway sends that same model name to the next credential and expects the provider to serve it. So a chain helps when its entries overlap on the models you call: an Anthropic credential with a Bedrock credential that serves Anthropic models, or two OpenAI accounts in different regions. The gateway skips a credential that cannot serve the resolved model. It does not call it. When the model name tells the gateway which provider owns it, through an alias, the `provider/model` form, or a known bare name, the chain visits only that provider's credentials on the VK. If the name tells the gateway nothing, the gateway walks the chain as configured, and the providers' own answers decide the outcome. A VK narrowed to specific providers is enforced here too. If the model resolves to a provider that the key cannot use, the gateway refuses with `model_not_allowed`. The refusal names the model and tells the key's owner to widen the key's provider access. ## Streaming Fallback behaviour differs based on when failure occurs: * **Before first chunk emits** → transparent fallback. The stream-setup call (`bifrost.ChatCompletionStreamRequest`) walks the chain the same way non-streaming dispatch does; the client sees a single stream from whichever slot accepted the request. `X-LangWatch-Fallback-Count` reports the skipped slot count. * **After first chunk has streamed** → **no mid-stream fallback**. The gateway emits a terminal `event: error` frame (with `code: upstream_mid_stream_failure`) and closes the connection. The client may retry; a fresh request would then re-walk the chain. This split is deliberate, splicing chunks from two providers would produce an inconsistent response with mismatched tool-call ids and accumulated-content replay. It's a hard "no" per contract and enforced by a byte-exact assertion on the SSE error frame shape so future refactors can't accidentally break it. See [Streaming → pre-connection fallback, mid-stream failure](/docs/ai-gateway/streaming#pre-connection-fallback-transparent) for the exact frame bytes and a worked example. ## Observing the chain in traces Fallback attribution today: * Prometheus counter `gateway_provider_attempts_total{credential_id, outcome}` increments once per attempt, with `outcome` in `success | fallback_success | retryable_5xx | not_found | rate_limit | timeout | network | circuit_open | non_retryable | chain_exhausted | context_done`. * Response header `X-LangWatch-Fallback-Count: N`, how many fallbacks were attempted before success. * Request-id correlation via `X-LangWatch-Gateway-Request-Id`, join the metric + log line back to the specific trace. Per-attempt nested spans (`langwatch.fallback.attempt`, `.reason` attrs) are a v1.1 observability follow-up. In v1, the counter + header give you the aggregate picture; per-attempt reasoning lives in the gateway log line for that request-id. If **all** attempts in the chain fail, the gateway returns the last provider's error envelope mapped to the OpenAI-compatible shape (`provider_error` or `upstream_timeout` as the `type`). ## Circuit breaker Each provider has an independent circuit breaker with a sliding window: | Default | Meaning | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | **Window:** 30 s | Failure events within the last 30 s count toward the open threshold | | **Threshold:** 10 failures | 10 failures in the window open the circuit | | **Open cooldown:** 60 s | Circuit stays open for 60 s; skipped on new requests regardless of fallback order | | **Half-open probe:** 1 request | After cooldown, a single probe is let through; if it succeeds, circuit closes; if it fails, another 60 s of open | Override at the service level: | Env var | Default | | ------------------------------- | ------- | | `LW_GATEWAY_CIRCUIT_WINDOW_S` | `30` | | `LW_GATEWAY_CIRCUIT_THRESHOLD` | `10` | | `LW_GATEWAY_CIRCUIT_COOLDOWN_S` | `60` | **Per-replica, not shared**: each gateway replica maintains its own breaker state. This is deliberate, under a large-scale outage, N replicas rediscovering the recovered provider independently is resilient; depending on Redis for breaker consensus is not. Circuit state is emitted as a Prometheus gauge `gateway_circuit_state{credential_id}`, one series per credential slot. The state is encoded in the gauge **value**, not in a label: `0` closed, `1` open, `2` half-open. Alert on `gateway_circuit_state == 1 for 5m` to catch real provider outages (as distinct from transient blips). When a request hits a circuit that's currently `open`, the gateway skips to the next entry in the fallback chain immediately, no wasted round-trip to a provider we already know is down. ## Sizing the chain Returns decrease after 3 entries. Each extra attempt spends one more provider round-trip, one after the other. With the default `max_attempts=3`, the worst case is three upstream calls in a row. The client has usually stopped waiting before the last one answers. Use chains of 2 to 3 entries for latency-sensitive traffic. Lower `max_attempts` if you need a tighter total budget. Keep longer chains for batch and offline work, where wall-clock time is cheap.