Troubleshooting

Documentation Index

Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

401 invalid_api_key

• Every request from one client returns 401.
• A VK that worked yesterday suddenly doesn’t.

# 1. Is the VK format right?
echo "$LANGWATCH_VK" | grep -E '^lw_vk_(live|test)_[0-9A-HJKMNP-TV-Z]{26}$' || echo "malformed"

# 2. Is it actually getting through?
curl -sD- -o/dev/null -H "Authorization: Bearer $LANGWATCH_VK" \
  https://gateway.langwatch.ai/v1/models

1. Env var expansion failed — $LANGWATCH_VK is empty at runtime. Check your deploy config.
2. VK was rotated; client still holds the old secret. Update to the new one.
3. VK was revoked. Status in UI → Gateway → Virtual Keys.
4. You pasted the prefix (lw_vk_live_01HZX...) instead of the full secret. The secret is always 40 chars total.
5. Rare: the gateway’s auth cache is stale. Should resolve within 30 s via /changes. If persistent, check control-plane /api/internal/gateway/changes is reachable.

​

403 model_not_allowed

• Specific models return 403, others work.
• New model you just added to the provider account is rejected.

curl -s -H "Authorization: Bearer $LANGWATCH_VK" \
  https://gateway.langwatch.ai/v1/models | jq -r '.data[].id' | sort

​

402 budget_exceeded

• Requests start failing mid-day after working all morning.
• Only specific users see it; others don’t.

langwatch gateway-budgets list --format json | \
  jq '.[] | select((.spent_usd | tonumber) >= (.limit_usd | tonumber) * 0.9)'

• Raise the cap: langwatch gateway-budgets update <id> --limit 1000.
• Switch the VK to a bigger-headroom scope.
• Change on_breach from block to warn if you just want visibility, not enforcement (but then spend is uncapped — be sure).

​

429 rate_limit_exceeded

• Bursty traffic 429s; steady traffic succeeds.
• Rate-limit headers in the response body mention tpm/rpm.

• error.type = rate_limit_exceeded + X-LangWatch-RateLimit-Dimension: rpm|rpd + Retry-After: <seconds> — gateway-level VK cap. The dimension header names which ceiling fired so dashboards can filter.
• type: rate_limit_exceeded WITH X-LangWatch-Provider: openai and no X-LangWatch-RateLimit-Dimension — upstream quota hit.

• If gateway-level: langwatch virtual-keys update <id> --config-json '{"rate_limits": {"rpm": 60, "rpd": 100000}}' to raise the cap, or {"rpm": null, "rpd": null} to disable. Ceiling changes take effect on the next request (limiter invalidates on every bundle refresh).
• If upstream: add a fallback provider to the VK chain, or request quota from the upstream.

​

403 tool_not_allowed

• Agent suddenly can’t run shell.exec / filesystem.write / specific tools.
• Was working; now isn’t.

• Intended policy — agent must not have this tool. Stop using it.
• Policy added in error — edit policy_rules.tools.deny on the VK.

​

503 service_unavailable / 504 upstream_timeout

• Intermittent 503/504 but NOT from upstream (no X-LangWatch-Provider header).
• Gateway /readyz starts flapping.

curl -s https://gateway.langwatch.ai/readyz | jq

• control_plane: error: timeout ... → control-plane degraded; gateway is refusing traffic correctly.
• auth_cache_bootstrap: error: ... → startup failed; pod will be restarted.
• outbox_writer: stale ... → budget debits aren’t flushing; budgets are stale.

• If control plane is down, wait / investigate control plane.
• If single pod, rolling-restart that pod — scheduler will replace.
• If fleet-wide, check LB + DNS before assuming gateway fault.

​

502 provider_error after fallback exhaustion

• X-LangWatch-Fallback-Count ≥ chain length.
• Multiple providers all failing at once.

• Real multi-provider incident: wait it out, post status page, shed traffic to queues.
• Credential issue: rotate the failing provider credentials (Gateway → Providers).

​

Cache-discount not applying (Anthropic)

• cache_read_input_tokens is always 0 on /v1/messages responses.
• Anthropic bill doesn’t show 90% cache discount.

# Is VK cache.mode set to 'respect'?
langwatch virtual-keys get <id> --format json | jq '.config.cache'

​

Request succeeded but no trace in my inbox

• /v1/chat/completions returned 200 with a real model response.
• Response carried X-LangWatch-Request-Id.
• But the LangWatch Traces screen in your project is empty (or doesn’t show this request).

# 1. Is the gateway configured to export spans at all?
# In the gateway pod:
echo "$GATEWAY_OTEL_DEFAULT_ENDPOINT"   # must be non-empty
echo "$GATEWAY_OTEL_DEFAULT_AUTH_TOKEN" # must be non-empty when endpoint is LangWatch's /api/otel

# 2. Is the exporter reaching the ingest?
kubectl logs <gateway-pod> | grep -E 'otlp|exporter'
# Look for:
#   - "exporter: export succeeded"                 -> spans left the gateway
#   - "status=401"                                 -> auth-token rejected at ingest
#   - "status=404"                                 -> endpoint URL is wrong (missing /v1/traces suffix — recent fix)
#   - "no endpoint configured"                     -> env var wasn't picked up

1. You’re still on a static token instead of the per-project resolver. The gateway resolves the X-Auth-Token from the VK’s owning project on every bundle refresh — each tenant’s spans route to their own inbox automatically. If you’ve hand-set GATEWAY_OTEL_DEFAULT_AUTH_TOKEN in your pod env, it takes precedence over the per-project token and every project’s spans land in whatever project that static token points at. Unset it and let the resolver do its job.
   Default dev setup needs no config. The gateway defaults GATEWAY_OTEL_DEFAULT_ENDPOINT to http://localhost:5560/api/otel and uses the per-project token from the bundle. pnpm dev and make run-dev Just Work — no env override needed. Override the endpoint only for prod/self-hosted.
2. Endpoint is empty. GATEWAY_OTEL_DEFAULT_ENDPOINT unset = no exporter wired. Spans are created in-pod and flushed to a no-op channel. Helm umbrella chart defaults this to http://langwatch-app:5560/api/otel — if you’ve overridden to empty, you’ve disabled trace export.
3. Auth token 401s at the ingest. /api/otel rejects unauthenticated or expired tokens with 401; the gateway exporter logs the rejection but the hot path is already done. Rotate the token, update the Secret, roll the gateway pod.
   OTLP 401/403 noise during the first 30s of startup is suppressed by design. The gateway’s OTel startup-error handler swallows transient transport errors (401, 403, connection-refused, no-such-host) for a 30-second grace window OR until the first successful export — whichever comes first. This avoids logspam during cold-boot races where the gateway is up before the control plane has minted the OTLP token. If you see OTLP 401 floods AFTER that grace window, the control-plane auth pipeline isn’t minting tokens — first-line check is LW_GATEWAY_INTERNAL_SECRET parity between the gateway and control plane (mismatched HMAC secret silently breaks the /resolve-key handshake that mints the OTLP token).
4. Endpoint URL 404s. Before the auto-suffix fix, an endpoint like http://langwatch-app:5560/api/otel was treated as the full URL; the Go OTLP exporter hit /api/otel and got 404 (the receiver is at /api/otel/v1/traces). Recent gateway builds append /v1/traces automatically when missing. Bump to the latest image.
5. Sampler dropped the span. Gateway spans are sampled at 1.0 by default — if you’ve tuned GATEWAY_OTEL_SAMPLER_ARG below 1.0 for cost, spot-checks will miss by design.
6. Trace landed but under a different project filter. Check that the Traces screen is scoped to the same project as the auth token’s project and that the Origin = Gateway quick-filter is either off or explicitly selected (the default view excludes non-Gateway origins when “Gateway” is active — and vice-versa).

​

auth_cache_refresh_transport_failure WARN spew in logs

• Sustained WARN-level lines reading auth_cache_refresh_transport_failure vk_id=... error=... error_class=transport new_soft_expires_at=....
• Optionally accompanied by INFO auth_cache_serve_stale lines on the same vk_ids.
• Customers report no rejection — traffic flows normally.

1. Check whether the control plane is actually unreachable (Kubernetes events, control-plane pod logs, /api/health from inside the gateway pod).
2. If it is, no operator action on the gateway side — wait for control-plane recovery and the WARN spew will stop on the next successful refresh (followed by a DEBUG auth_cache_refresh_success line per VK).
3. If you start seeing auth_cache_hard_evict reason=hard_cap_exceeded ERROR lines, the grace cap has been crossed and customers are now being rejected — see Production runbook → Recipe 8 for the operator playbook + how to extend the cap if the outage is ongoing.
4. To disable stale-while-error entirely (regulated deployments where stale-served bundles are unacceptable), set LW_GATEWAY_AUTH_CACHE_HARD_GRACE=0s. Refresh failures past JWT exp will then evict immediately and customers will see 401 invalid_api_key for the duration of the outage.

​

Trace-id propagation not working

• X-LangWatch-Trace-Id on response doesn’t match the traceparent you sent.
• Gateway spans show up as new root traces instead of children of your app trace.

curl -sD- -o/dev/null \
  -H "Authorization: Bearer $LANGWATCH_VK" \
  -H "traceparent: 00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-1111111111111111-01" \
  -H "Content-Type: application/json" \
  -X POST https://gateway.langwatch.ai/v1/chat/completions \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"ping"}],"max_tokens":4}' | \
  grep -i x-langwatch-trace-id

• SDK isn’t forwarding the traceparent header. Check default_headers / defaultHeaders in your client.
• Your trace propagator isn’t active. Use langwatch.get_gateway_headers() / getGatewayHeaders().
• Proxy / LB between client and gateway is stripping the header. Curl directly to the gateway DNS to bypass.

​

Streaming hangs / drops mid-response

• Chat assistant produces half a response then nothing.
• No error, just silence; client eventually times out.

• langwatch.stream.chunk_count — how many chunks reached the gateway.
• langwatch.stream.terminated_at_chunk — present when gateway observed mid-stream failure.

​

”My config override isn’t taking effect”

• You set an env var via Helm or values.yaml, deployed, but the gateway behaves as if the default were still in effect.
• kubectl exec + env shows the var is set, but the behaviour doesn’t match.

POD=$(kubectl get pod -n langwatch -l app=langwatch-gateway -o jsonpath='{.items[0].metadata.name}')

# Dump the env the pod sees — anything set via Helm, ConfigMap, or Secret refs
kubectl exec -n langwatch "$POD" -- env | grep -E '^(LW_GATEWAY|CUSTOMER_TRACE_BRIDGE|OTEL|SERVER|LOG)_' | sort

1. Stale pods — you bumped values.yaml but haven’t restarted. kubectl rollout restart deploy/langwatch-gateway.
2. The VK’s bundled config overrides the pod-level default (e.g. VK has its own cache.mode). Check the VK config via langwatch vk get <id>.
3. The control plane hasn’t propagated the bundle change. Wait 30 s for the /changes long-poll cycle, or restart gateway pods.

1. The env var is set on the wrong container in the deployment (e.g. on an init container).
2. Helm value was set on the wrong key (gateway.env.X vs gateway.X).
3. ConfigMap rendered the value as a YAML scientific-notation number (a known gotcha for int64 fields) — check kubectl describe configmap.

​

413 payload_too_large

• A request with a large prompt / large base64 image / long tool-call argument list returns 413.
• Response body: {"error":{"type":"payload_too_large","message":"request body exceeds 33554432 bytes"}}.

1. Compress / resize large base64 images on the client before including them in the payload. A 4K image base64-encoded is ~6 MB on its own; multiple full-res images plus a big system prompt can push past 32 MiB.
2. Trim long tool-call argument lists or chunk the request across multiple completions.
3. If your traffic legitimately exceeds 32 MiB (multi-image vision with 1M-context models), raise SERVER_MAX_REQUEST_BODY_BYTES on the gateway deployment. Coordinate with ops — raising the cap reduces attack-surface cheapness, so the higher value should be deliberate and scoped to the impacted deployment.

​

Pod never becomes ready after a deploy

• kubectl rollout status hangs, pod cycles through CrashLoopBackOff or stays in Running with /startupz returning 503.
• First customer request would have failed, but you’re catching it at deploy.

POD=$(kubectl get pod -n langwatch -l app=langwatch-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n langwatch "$POD" | grep -E 'dns resolution failed|tcp dial failed'

• dns resolution failed: lookup api.openai.com: no such host — the NetworkPolicy is blocking kube-system :53 egress. Check the DNS rule is FIRST in spec.egress (order matters in some CNIs) and the kube-system selector matches your cluster’s DNS namespace (some clusters use kube-dns, others a custom namespace).
• tcp dial failed: dial tcp <ip>:443: connect: connection refused — DNS works but provider egress is blocked. Your egressToProviders override is too narrow; check the resolved IP is in an allowed CIDR, or widen back to the default (any-IP except RFC1918).
• tcp dial failed: i/o timeout — firewall (VPC security group / on-prem) is dropping the packet silently. DNS worked, NetworkPolicy allowed it, but an upstream network layer did not. Check your cloud provider’s security groups / NAT gateway rules.

​

First-request latency spike

• First request after pod scale-up takes 3-5 s; subsequent requests take 200 ms.
• Autoscaler adds replicas during traffic bursts.

• Enable Redis L2 (GATEWAY_REDIS_URL) — new replicas warm from Redis instead of control plane.
• Ensure pod readiness probe waits for the long-poll loop to be running (/readyz handles this correctly by default).
• GATEWAY_CACHE_BOOTSTRAP_ALL_KEYS=true is a planned v1.1 enhancement that would pre-warm the L1 LRU with every active VK on startup; the flag is reserved in env wiring but currently has no Go-side implementation, so setting it does not help today.

​

Debug checklist — common patterns

1. Take the X-LangWatch-Request-Id from the failing request → LangWatch search.
2. Compare X-LangWatch-Provider to the VK’s configured primary — if different, fallback fired.
3. Check X-LangWatch-Cache response header: hit / miss / bypass / force (the last is v1.1 — deferred in v1).
4. Gateway pod logs filtered on request_id=grq_... for structured entries.
5. Control-plane logs filtered on vk_id for authorization decisions.

​

503 service_unavailable — guardrail upstream

• Requests that previously worked return 503 with type=service_unavailable when a guardrail evaluator fails and the VK is fail-closed (the default).
• The LangWatch evaluator dashboard shows errors or timeouts on evaluators attached to the VK.
• Metric gateway_guardrail_verdicts_total{verdict="block"} climbs on the affected direction (request / response) — or {verdict="fail_open"} if the VK opts in.

# Which guardrails are attached to the VK?
langwatch virtual-keys get <vk_id> --format json | jq '.config.guardrails'

# Is the fail-open toggle set (per direction)?
langwatch virtual-keys get <vk_id> --format json | \
  jq '{req: .config.guardrails.request_fail_open, resp: .config.guardrails.response_fail_open}'

1. Restore the evaluator service. Check LangEvals pod / webhook endpoint health. The most common causes are (a) evaluator service OOM-killed by a long-running evaluator, (b) evaluator secret (LLM API key) expired.
2. If the evaluator is optional (best-effort redaction, nice-to-have), flip fail-open for that direction:
   langwatch virtual-keys update <vk_id> --config-json \
     '{"guardrails": {"response_fail_open": true}}'
   
   Warning: fail-open means requests pass through ungoverned when evaluators are down. Only do this for low-stakes traffic.
3. For stream_chunk guardrails, the 50 ms per-chunk budget is fail-open by contract (no opt-in needed) — blocking the user’s stream on a slow policy service is worse than pass-through. Check langwatch.guardrail.stream_chunk_fail_open=<reason> trace attributes for why chunks are being passed through.

​

503 service_unavailable — policy_rules compile failure

• All requests through one VK return 503 with type=service_unavailable.
• Gateway WARN log line policy_rules_compile_failed (dispatcher.go:253) — names the dimension (tools, mcp, urls, models) and the broken pattern.
• Follow-up WARN line policy_rules_broken on every subsequent request until the policy is fixed.

langwatch virtual-keys get <vk_id> --format json | jq '.config.policy_rules'

• Unescaped parentheses — use \\( \\)
• Unterminated character classes — [abc missing ]
• Backrefs \1 — RE2 doesn’t support backreferences (by design, for O(n) guarantees). Use alternation (a|b) instead.

# Quick RE2 syntax check
echo 'test' | docker run -i --rm golang:1.22 go run -e 'package main
import ("fmt"; "os"; "regexp")
func main() { _, err := regexp.Compile(os.Args[1]); if err != nil { fmt.Println(err); os.Exit(1) }; fmt.Println("ok") }' '<your_pattern>'

​

504 provider is required

• Requests intermittently return 504 provider is required for a VK that worked fine seconds before.
• Direct curl with a known-good model name fails the same way.

• Restart the gateway pod (kubectl rollout restart deploy/langwatch-gateway in prod, pkill -f "service aigateway" && make service svc=aigateway in dev). A fresh bundle fetch resolves it.
• If recurring under steady load, file the gateway log line + repro shape — it points at a Bundle resolver TTL/refresh bug.

​

504 HTML response received from provider

• 504 with the body containing event: response.created\ndata: {…} or just plain HTML fragments.
• Affects coding-agent CLIs (codex, claude-code) more than direct API clients.

1. Body too large for the upstream’s edge. Some providers return HTML 4xx pages instead of JSON when the request body exceeds their per-request tolerance. Coding agents can balloon request bodies past 50 KiB with rich tool schemas + system prompts. Fix: reduce body — use claude --bare --disable-slash-commands to strip parent skills + plugins; for codex, lower model_reasoning_effort and avoid stuffing large instructions blocks.
2. Upstream auth failure. Provider returned an HTML 401/403 page rather than JSON. Verify the gateway VK’s underlying provider credential is valid (see Settings → Model Providers).
3. Upstream API rate limit / outage. Provider is degraded; falls back via the VK chain if configured.
4. The streaming response was misclassified as non-streaming. Was historically the cause for codex 0.122+ on /v1/responses — stream:true lived past the gateway’s 32 KiB peek window and PeekStream(peek) returned false. Fixed by widening the peek window for /v1/responses to 256 KiB.

​

504 failed to unmarshal response from provider API

• Gateway returns generic 504 with “failed to unmarshal response from provider API”.
• The actual upstream error from the provider is hidden behind the generic message.

​

Boot fails with AI Gateway secrets partial config

• Any LangWatch entry point (pnpm dev, pnpm start, workers, prisma:*, CLI scripts) refuses to start.
• Loud banner in the boot log naming a subset of LW_VIRTUAL_KEY_PEPPER, LW_GATEWAY_INTERNAL_SECRET, LW_GATEWAY_JWT_SECRET as missing.

# Whichever the banner names as missing
echo "LW_VIRTUAL_KEY_PEPPER=$(openssl rand -hex 32)" >> langwatch/.env
echo "LW_GATEWAY_INTERNAL_SECRET=$(openssl rand -hex 32)" >> langwatch/.env
echo "LW_GATEWAY_JWT_SECRET=$(openssl rand -hex 32)" >> langwatch/.env

​

See also

• API: Errors — full error-type enum and HTTP status mapping.
• Observability — what every span attribute means.
• Health Checks — interpreting /readyz output.
• Cookbook: CI smoke test — automate validation.
• Cookbook: Production runbook — on-call playbook (graceful drain, outbox backlog, pprof diagnostics).
• Cookbook: Grafana dashboard — importable JSON to visualise every metric referenced on this page.