Production runbook

Documentation Index

Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

Prerequisites

# Go toolchain with pprof (matches gateway's Go version)
go version  # expect 1.26+

# Pick one pod to focus on
POD=$(kubectl get pod -n langwatch -l app=langwatch-gateway -o jsonpath='{.items[0].metadata.name}')

# Open the tunnel in one terminal and leave it running
kubectl port-forward -n langwatch "$POD" 6060:6060

​

Enabling pprof via Helm

gateway:
  admin:
    addr: "127.0.0.1:6060"   # default — reachable only via kubectl port-forward
    # addr: ""               # disable pprof entirely (advised for compliance-regulated envs)

​

Recipe 1 — p99 latency spike

# 30-second CPU profile while the spike is live
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Inside pprof:
(pprof) top20
(pprof) web     # opens an SVG flame graph in your browser

• A single function using > 50% of CPU that’s not one of: tls.conn.Handshake, net/http.(*conn).serve, json.Decoder.Decode. Those are expected under load.
• Lock contention on internal/auth or internal/fallback — hot in sync.(*Mutex).Lock. Usually means the L1 cache is evicting faster than it’s filling; consider raising LW_GATEWAY_AUTH_CACHE_L1_SIZE.
• RE2 compilation in internal/blocked on every request — means the bundle isn’t caching compiled regexes. Check for frequent /changes churn (revision bumps on every request ≠ normal).

​

Recipe 2 — Goroutine leak

# Live goroutine dump as SVG
go tool pprof -svg http://localhost:6060/debug/pprof/goroutine > /tmp/goroutines.svg
open /tmp/goroutines.svg

# Or aggregate counts by creation site
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -100

• Hundreds of goroutines parked in chan receive inside internal/fallback.Walk — means a fallback attempt is hanging on a context that never cancels. Check LW_GATEWAY_UPSTREAM_TIMEOUT_MS.
• Goroutines stuck in internal/guardrails.CheckChunk — likely a guardrail evaluator that never returns and exceeds the 50 ms budget. Check the evaluator service logs.
• Streaming goroutines (internal/dispatch.streamSSE) that outlive their request context — client disconnect without ctx.Done() firing usually points at a missing close somewhere.

​

Recipe 3 — Memory growth

# Live heap snapshot
go tool pprof http://localhost:6060/debug/pprof/heap

# Inside pprof:
(pprof) top20 -cum
(pprof) list internal/auth    # source-level attribution

• internal/auth.(*Cache).Put holding more than ~1 MB per cached bundle — unusual, a bundle should be ≤ 50 KB. Oversized policy_rules.urls.allow with thousands of entries can trigger this.
• internal/dispatch buffered responses — if streaming responses are being accumulated instead of flushed per-chunk, every request consumes full response size. Check for bufio.NewWriter wrapping a streaming writer anywhere.
• Outbox shelving (LW_GATEWAY_BUDGET_DEBIT_SHELF_BYTES) — a long control-plane outage can push debits to disk. Check the outbox directory size.

​

Recipe 4 — Mutex or block profiling for contention

# Top mutex holders
go tool pprof http://localhost:6060/debug/pprof/mutex

# Top blockers (I/O, channels, locks)
go tool pprof http://localhost:6060/debug/pprof/block

• Contention on internal/ratelimit.(*Bucket).Allow — means a VK’s RPM is bursting past the token-bucket refill rate and every request is waiting. Raise RPM or investigate the caller.
• Contention on the L1 auth cache — see Recipe 1.

​

Recipe 5 — Allocation churn (GC pressure)

# Alloc-only heap (who's creating the most garbage)
go tool pprof http://localhost:6060/debug/pprof/allocs

(pprof) top20 -cum

• JSON encoding of large /v1/messages requests — expected, but if it dominates consider enabling LW_GATEWAY_MAX_BODY_BYTES enforcement to reject pathologically large bodies earlier.
• Per-request compilation of the same regex — should never happen; if it does, Lane A has a caching regression.

​

Recipe 6 — Debit outbox backlog

# 1. Fill percentage right now
kubectl exec -n langwatch deploy/langwatch-gateway -- \
  wget -qO- http://127.0.0.1:5590/metrics | \
  grep -E '^gateway_budget_debit_outbox_(depth|capacity)'

# 2. Last 15 min of flush failures (delta)
# Graph this in Grafana with:
#   rate(gateway_budget_debit_outbox_flush_failures_total[5m])

# 3. 4xx drops — any non-zero delta is a page
#   increase(gateway_budget_debit_outbox_4xx_drops_total[15m])

• Only depth is rising, flush_failures is 0: normal burst. Fill-pct will self-heal when traffic subsides. Only page if sustained > 80% for > 10m.
• flush_failures > 0 and depth is rising: control plane is unavailable or slow. Check control-plane /api/internal/gateway/budget handler latency and error rate. Debits are safe (re-enqueued); they’ll drain once the control plane recovers.
• 4xx_drops > 0: immediate page. Likely causes, in order of probability:
  1. HMAC secret drift between gateway and control plane (LW_GATEWAY_INTERNAL_SECRET mismatch after rotation).
  2. Payload schema drift (gateway and control plane deployed with incompatible contract versions).
  3. Rare: control plane returning 4xx for a specific tenant (archived organization, deleted project).
• dropped_total > 0: the ring filled up and the oldest pending events were evicted before they could flush. Always follows a prolonged control-plane outage. Lost debits are unrecoverable — document the outage window for post-hoc spend reconciliation.

• rate(gateway_budget_debit_outbox_4xx_drops_total[5m]) > 0 → page (silent data loss).
• rate(gateway_budget_debit_outbox_flush_failures_total[5m]) > 0 for 5m → warn.
• (max(gateway_budget_debit_outbox_depth) / max(gateway_budget_debit_outbox_capacity)) > 0.5 for 5m → warn.

​

Recipe 7 — Stuck drain

# 1. Which pod is stuck, and how many handlers are hanging?
kubectl get pod -n langwatch -l app=langwatch-gateway \
  --field-selector=status.phase=Running -o wide | grep Terminating

# 2. Goroutine dump — stuck handlers will show up in Go runtime state
POD=<terminating pod name>
kubectl port-forward -n langwatch "$POD" 6060:6060 &
go tool pprof -top http://localhost:6060/debug/pprof/goroutine

# 3. Find the handler that's not returning
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | \
  grep -A 20 'internal/dispatch.*streamSSE\|internal/auth\|internal/guardrails'

1. Upstream dial hanging without a deadline. Streaming fallback into a dead region that never returns TLS handshake. LW_GATEWAY_UPSTREAM_TIMEOUT_MS should be < shutdown.timeout; if it is, the handler should cancel on its own.
2. Guardrail evaluator hanging past its budget. pre/post have a guardrail.preTimeout / postTimeout of 1500 ms, but a misconfigured evaluator can still hang if it doesn’t respect context cancellation. Check the evaluator service’s own SLO.
3. A breaker with open state but no surrounding deadline. Rare; closed previously, but worth ruling out if the stack shows internal/circuit waiting.
4. Slow custom middleware. If you’ve forked the gateway and added middleware that does I/O without context propagation, that’s where to look first.

​

Recipe 8 — Control-plane outage / stale-while-error

# 1. Are stale-serve INFO logs firing? (= grace is active and serving traffic)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=200 | \
  grep auth_cache_serve_stale

# 2. Are transport-class refresh failures spewing? (= control plane unreachable)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=200 | \
  grep auth_cache_refresh_transport_failure

# 3. Are hard evictions firing? (= grace exhausted, rejection mode)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=500 | \
  grep auth_cache_hard_evict

• Requests against any VK that resolved successfully before the outage continue to work transparently.
• Requests against any VK never seen by this pod (cold) still fail — the gateway has no bundle to fall back to. Today’s mitigation is Redis L2 (GATEWAY_REDIS_URL): HPA-scaled pods inherit the warm set from L2 even while the control plane is unreachable. (GATEWAY_CACHE_BOOTSTRAP_ALL_KEYS=true is a planned v1.1 enhancement to also pre-warm L1 from a /bootstrap snapshot on startup; the flag is reserved in env wiring but has no Go-side implementation today, so setting it is currently a no-op.)
• Requests against any VK whose JWT was revoked just before the outage but the revocation /changes event hadn’t propagated yet — these stay served until the cache entry crosses its hard cap. Acceptable trade-off for the grace; auth rejections from a healthy CP still evict instantly via auth_cache_hard_evict reason=auth_rejection.

# Fire if hard_evict reason=hard_cap_exceeded appears more than N times in 5 min
# Indicates the outage exceeded the configured grace and customers are now being rejected.
kubectl logs -n langwatch -l app=langwatch-gateway --since=5m | \
  grep -c 'auth_cache_hard_evict.*reason=hard_cap_exceeded'

​

Graceful degradation — what survives what

​

Writing findings back

1. Grab the .pb.gz with go tool pprof -symbolize=remote -proto http://localhost:6060/debug/pprof/heap > /tmp/heap.pb.gz.
2. Attach it to the issue along with:
   • Pod name + image digest (kubectl get pod -n langwatch -o yaml | grep image).
   • X-LangWatch-Request-Id from one exemplar bad request.
   • kubectl top pod output around the spike.
3. For urgent escalation, post in #ai-gateway-support with the request id.

​

See also

• Config → Admin / operator endpoints — env-var reference.
• Health Checks — /readyz output to interpret alongside pprof.
• Troubleshooting — symptom-first diagnostic index for non-pprof issues (401 / 403 / 429 / 502).
• Prometheus alerts — alert rules that tell you when to reach for this runbook.