> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Production runbook

> Diagnose p99 spikes, goroutine leaks, memory growth, and other "why is the gateway slow?" incidents using the loopback-bound pprof admin listener.

When the gateway is slow, stuck, or chewing memory, and the LangWatch trace for a specific bad request has already been checked, this runbook is the next stop. Every recipe here relies on the pprof admin listener, which is bound to `127.0.0.1:6060` by default and therefore never reachable from outside the pod.

<Note>**Don't expose the admin port without a token.** `GATEWAY_ADMIN_ADDR` binds to loopback by default. If you genuinely need direct (non-port-forward) access, e.g. non-k8s deploys, or from a corporate VPN, you MUST also set `GATEWAY_ADMIN_AUTH_TOKEN`. The gateway refuses to start otherwise. See [Helm → Admin listener](/docs/ai-gateway/self-hosting/helm#admin-listener) for the three deployment postures. `kubectl port-forward` remains the simplest option for k8s, it tunnels through the API server and is auditable in Kubernetes audit logs.</Note>

## Prerequisites

One-time setup on your operator laptop:

```bash theme={null}
# Go toolchain with pprof (matches gateway's Go version)
go version  # expect 1.26+

# Pick one pod to focus on
POD=$(kubectl get pod -n langwatch -l app=langwatch-gateway -o jsonpath='{.items[0].metadata.name}')

# Open the tunnel in one terminal and leave it running
kubectl port-forward -n langwatch "$POD" 6060:6060
```

All subsequent `go tool pprof` commands target `http://localhost:6060`.

### Enabling pprof via Helm

The LangWatch Helm chart exposes the admin listener via a top-level `admin` stanza on the gateway sub-chart. The loopback-bound default is what you want in production, if you widen it to `0.0.0.0`, also set `admin.existingAuthSecretName` so the built-in bearer-token guard protects pprof. The gateway refuses to start in the bind-non-loopback-without-token configuration.

```yaml theme={null}
gateway:
  admin:
    addr: "127.0.0.1:6060"   # default — reachable only via kubectl port-forward
    # addr: ""               # disable pprof entirely (advised for compliance-regulated envs)
```

Shipped in current chart versions. Older chart versions pre-date the field, upgrade the chart before troubleshooting, or set the env var directly via `gateway.env.GATEWAY_ADMIN_ADDR`.

## Recipe 1: p99 latency spike

**Symptom:** `histogram_quantile(0.99, sum by (le) (rate(gateway_http_request_duration_seconds_bucket[5m])))` jumps from \~300 ms to several seconds. Traces show no single slow upstream.

**Diagnose:**

```bash theme={null}
# 30-second CPU profile while the spike is live
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Inside pprof:
(pprof) top20
(pprof) web     # opens an SVG flame graph in your browser
```

**What to look for:**

* A single function using > 50% of CPU that's not one of: `tls.conn.Handshake`, `net/http.(*conn).serve`, `json.Decoder.Decode`. Those are expected under load.
* Lock contention on `internal/auth` or `internal/fallback`, hot in `sync.(*Mutex).Lock`. Usually means the L1 cache is evicting faster than it's filling; consider raising `LW_GATEWAY_AUTH_CACHE_L1_SIZE`.
* RE2 compilation in `internal/blocked` on every request, means the bundle isn't caching compiled regexes. Check for frequent `/changes` churn (revision bumps on every request ≠ normal).

## Recipe 2: Goroutine leak

**Symptom:** `go_goroutines` climbs monotonically over hours, never GCs. Memory follows.

**Diagnose:**

```bash theme={null}
# Live goroutine dump as SVG
go tool pprof -svg http://localhost:6060/debug/pprof/goroutine > /tmp/goroutines.svg
open /tmp/goroutines.svg

# Or aggregate counts by creation site
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -100
```

**What to look for:**

* Hundreds of goroutines parked in `chan receive` inside `internal/fallback.Walk`, means a fallback attempt is hanging on a context that never cancels. Check `LW_GATEWAY_UPSTREAM_TIMEOUT_MS`.
* Goroutines stuck in `internal/guardrails.CheckChunk`, likely a guardrail evaluator that never returns and exceeds the 50 ms budget. Check the evaluator service logs.
* Streaming goroutines (`internal/dispatch.streamSSE`) that outlive their request context, client disconnect without `ctx.Done()` firing usually points at a missing `close` somewhere.

Fix options: restart the offending pod (workaround), or file a bug with the `/tmp/goroutines.svg` attached.

## Recipe 3: Memory growth

**Symptom:** RSS climbs from 200 MB to > 1 GB over a day. OOM eventually follows.

**Diagnose:**

```bash theme={null}
# Live heap snapshot
go tool pprof http://localhost:6060/debug/pprof/heap

# Inside pprof:
(pprof) top20 -cum
(pprof) list internal/auth    # source-level attribution
```

**What to look for:**

* `internal/auth.(*Cache).Put` holding more than \~1 MB per cached bundle, unusual, a bundle should be ≤ 50 KB. Oversized `policy_rules.urls.allow` with thousands of entries can trigger this.
* `internal/dispatch` buffered responses, if streaming responses are being accumulated instead of flushed per-chunk, every request consumes full response size. Check for `bufio.NewWriter` wrapping a streaming writer anywhere.
* Cached bundles never being evicted, a very large tenant count with long `HARD_GRACE` keeps every bundle resident. Check `gateway_auth_cache_size{tier="l1"}` against your expected VK count.

## Recipe 4: Mutex or block profiling for contention

**Symptom:** CPU is low, request rate is low, but latency is up. Suggests blocking, not computation.

Enable on a specific pod (requires a restart with `LW_GATEWAY_PPROF_BLOCK_RATE=1` and `LW_GATEWAY_PPROF_MUTEX_FRACTION=1` in the env, off by default because they have measurable overhead).

```bash theme={null}
# Top mutex holders
go tool pprof http://localhost:6060/debug/pprof/mutex

# Top blockers (I/O, channels, locks)
go tool pprof http://localhost:6060/debug/pprof/block
```

**What to look for:**

* Contention on `internal/ratelimit.(*Bucket).Allow`, means a VK's RPM is bursting past the token-bucket refill rate and every request is waiting. Raise RPM or investigate the caller.
* Contention on the L1 auth cache, see Recipe 1.

## Recipe 5: Allocation churn (GC pressure)

**Symptom:** `go_gc_pause_seconds_sum` is growing too fast; p99 spikes correlate with GC.

**Diagnose:**

```bash theme={null}
# Alloc-only heap (who's creating the most garbage)
go tool pprof http://localhost:6060/debug/pprof/allocs

(pprof) top20 -cum
```

**What to look for:**

* JSON encoding of large `/v1/messages` requests, expected, but if it dominates consider enabling `LW_GATEWAY_MAX_BODY_BYTES` enforcement to reject pathologically large bodies earlier.
* Per-request compilation of the same regex, should never happen; if it does, Lane A has a caching regression.

## Recipe 6: Budget spend not accumulating

**Symptom:** Budget scopes never approach their limit even though traffic is flowing, or `gateway_budget_blocks_total{scope}` stays flat on a scope you expect to be capped.

The gateway does not send debits. Cost is captured as attributes on the OTel span the gateway emits; the control plane's trace-fold reactor (`gatewayBudgetSync.reactor.ts`) reads finalised spans and writes the ClickHouse ledger rows. Budget enforcement on the gateway is a purely local precheck against the budget snapshot baked into the cached bundle, so spend only moves when the whole chain runs:

```
gateway request -> OTel span (cost attrs) -> collector -> langwatch-workers
  -> trace-fold reactor -> ClickHouse ledger -> bundle snapshot -> gateway precheck
```

**Diagnose, in order:**

| Check                         | How                                                                                                                     | If it fails                                                                         |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Spans leaving the gateway     | `sum(rate(gateway_http_requests_total[5m]))` against your collector's received-spans counter                            | OTLP export is broken; see [Observability](/docs/ai-gateway/observability)               |
| Workers running               | `kubectl -n langwatch get deploy langwatch-workers`                                                                     | `workers.enabled: false` means the ledger is never written and hard caps never fire |
| Ledger advancing              | Query `gateway_budget_ledger_events` in ClickHouse for the scope                                                        | Reactor is stalled or the span carried no cost attrs                                |
| Gateway seeing the new totals | Watch `gateway_internal_rtt_seconds{endpoint="changes"}` and `gateway_control_plane_requests_total{endpoint="changes"}` | The bundle is not refreshing, so the gateway is prechecking against stale spend     |

**Operator signal:** `gateway_budget_blocks_total{scope}` is the counter that tells you enforcement actually fired. A scope configured with `on_breach: block` whose ledger is past the limit but whose block counter is flat means the gateway has not received a refreshed bundle yet.

<Note>Because the precheck is local and permissive, a stale snapshot allows requests through rather than rejecting them. Overspend during a workers outage is expected and reconciles once the ledger catches up.</Note>

## Recipe 7: Stuck drain

**Symptom:** A pod stays in `Terminating` for the full `terminationGracePeriodSeconds`, then gets SIGKILLed. In-flight requests ended abruptly. Alertmanager fires on the `gateway_draining` gauge being 1 for > grace.

The pod received SIGTERM but at least one request handler never returned before `shutdown.timeout` expired. The drain pipeline exposes this via two gauges:

| Metric                       | During healthy drain | During stuck drain        |
| ---------------------------- | -------------------- | ------------------------- |
| `gateway_draining`           | 1 → 0 at shutdown    | 1 until SIGKILL           |
| `gateway_in_flight_requests` | monotonically → 0    | flat at N > 0 for > grace |

**Diagnose:**

```bash theme={null}
# 1. Which pod is stuck, and how many handlers are hanging?
kubectl get pod -n langwatch -l app=langwatch-gateway \
  --field-selector=status.phase=Running -o wide | grep Terminating

# 2. Goroutine dump, stuck handlers will show up in Go runtime state
POD=<terminating pod name>
kubectl port-forward -n langwatch "$POD" 6060:6060 &
go tool pprof -top http://localhost:6060/debug/pprof/goroutine

# 3. Find the handler that's not returning
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | \
  grep -A 20 'internal/dispatch.*streamSSE\|internal/auth\|internal/guardrails'
```

**Common causes, in order of likelihood:**

1. **Upstream dial hanging without a deadline.** Streaming fallback into a dead region that never returns TLS handshake. `LW_GATEWAY_UPSTREAM_TIMEOUT_MS` should be \< `shutdown.timeout`; if it is, the handler should cancel on its own.
2. **Guardrail evaluator hanging past its budget.** `pre`/`post` have a `guardrail.preTimeout`, `postTimeout` of 1500 ms, but a misconfigured evaluator can still hang if it doesn't respect context cancellation. Check the evaluator service's own SLO.
3. **A breaker with `open` state but no surrounding deadline.** Rare; closed previously, but worth ruling out if the stack shows `internal/circuit` waiting.
4. **Slow custom middleware.** If you've forked the gateway and added middleware that does I/O without context propagation, that's where to look first.

**Fix:**

| Cause             | Action                                                                                                                                                                                             |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Upstream hang     | Lower `LW_GATEWAY_UPSTREAM_TIMEOUT_MS` below `shutdown.timeout`. Default 60 s, bump `shutdown.timeout` to 65 s + `terminationGracePeriodSeconds` to 80 s if you genuinely need 60 s upstream calls |
| Guardrail hang    | Raise evaluator SLO or set `guardrails.request_fail_open: true` on the VK to fall through on timeout                                                                                               |
| Custom middleware | Thread `r.Context()` through every I/O call                                                                                                                                                        |

**Temporary workaround:** bump `shutdown.timeout` + `terminationGracePeriodSeconds` to give the hanging request time to complete. Only appropriate while you diagnose the root cause, long grace periods slow down rolling deploys and make HPA scale-downs feel sluggish.

## Recipe 8: Control-plane outage, stale-while-error

**Symptom:** The LangWatch control plane is unreachable (deployment incident, DNS hiccup, network partition). The gateway's L1 auth cache is full of valid resolved-key bundles, but `/api/internal/gateway/resolve-key` is returning errors. Operator wants to know: are customers being rejected, or is the gateway riding through?

The gateway's auth resolver runs **stale-while-error** by default: when the cached entry's JWT crosses its natural expiry AND the control-plane refresh fails for **transport-class reasons** (network error, dial timeout, 5xx, connection refused, malformed/unparseable response, JWT verify failure), it bumps the soft expiry by `LW_GATEWAY_AUTH_CACHE_SOFT_BUMP` (default `5m`) and serves the cached bundle. This continues every refresh attempt up to the hard cap of `LW_GATEWAY_AUTH_CACHE_HARD_GRACE` past the JWT exp (default `6h`). The hard cap is deliberately generous, the soft-bump path runs on **every** refresh attempt without a successful response, so the hard cap is the true outage backstop, not a steady-state knob.

**Auth-class rejections**: explicit `401`, `403`, `404` from `/resolve-key`, bypass the grace window entirely and evict immediately. A revoked credential never gets stale-served.

**Diagnose:**

```bash theme={null}
# 1. Are stale-serve INFO logs firing? (= grace is active and serving traffic)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=200 | \
  grep auth_cache_serve_stale

# 2. Are transport-class refresh failures spewing? (= control plane unreachable)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=200 | \
  grep auth_cache_refresh_transport_failure

# 3. Are hard evictions firing? (= grace exhausted, rejection mode)
kubectl logs -n langwatch -l app=langwatch-gateway --tail=500 | \
  grep auth_cache_hard_evict
```

The three log lines form a ladder operators read in order:

| Log line                               | Level | Meaning                                                                                                                                                                                                            | Operator action                                                                                                                                       |
| -------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auth_cache_serve_stale`               | INFO  | Grace is active; cached bundle is being served past JWT exp                                                                                                                                                        | None, this is expected behaviour during outage. Includes `vk_id`, `stale_for`, `hard_grace_remaining`, `refresh_error_class`                          |
| `auth_cache_refresh_transport_failure` | WARN  | Each refresh attempt is failing; soft expiry bumped. Includes `error`, `error_class`, `new_soft_expires_at`                                                                                                        | Investigate control plane (this is the actual outage signal). Stop bumping once CP returns                                                            |
| `auth_cache_hard_evict`                | ERROR | Grace cap exceeded OR auth rejection. `reason` field disambiguates: `hard_cap_exceeded` (outage too long) vs `auth_rejection` (real bad-credential evict) vs `auth_rejection_async`, `hard_cap_exceeded_on_lookup` | If `hard_cap_exceeded`, customers start seeing 401s. Bump `LW_GATEWAY_AUTH_CACHE_HARD_GRACE` if outage is ongoing and you'd rather extend than reject |

**Customer-facing behaviour during the grace window:**

* Requests against any VK that resolved successfully **before** the outage continue to work transparently.
* Requests against any VK **never seen by this pod** (cold) still fail, the gateway has no bundle to fall back to. Today's mitigation is Redis L2 (`GATEWAY_REDIS_URL`): HPA-scaled pods inherit the warm set from L2 even while the control plane is unreachable. (`GATEWAY_CACHE_BOOTSTRAP_ALL_KEYS=true` is a planned v1.1 enhancement to also pre-warm L1 from a `/bootstrap` snapshot on startup; the flag is reserved in env wiring but has no Go-side implementation today, so setting it is currently a no-op.)
* Requests against any VK whose JWT was **revoked just before the outage** but the revocation `/changes` event hadn't propagated yet, these stay served until the cache entry crosses its hard cap. Acceptable trade-off for the grace; auth rejections from a healthy CP still evict instantly via `auth_cache_hard_evict reason=auth_rejection`.

**Tune:**

| Knob                                  | When to lower                                                                                                                                                                | When to raise                                                                                                                 |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `LW_GATEWAY_AUTH_CACHE_SOFT_BUMP`     | If you want refresh attempts to back off less aggressively (e.g. 1m for tighter retry density)                                                                               | If your control plane is consistently slow under load and you want to reduce refresh-attempt churn during partial degradation |
| `LW_GATEWAY_AUTH_CACHE_HARD_GRACE`    | If you have strict revocation-latency SLOs (compliance, regulated deployments) and would rather hard-fail than serve a long-stale bundle                                     | If your control plane has known multi-hour planned-maintenance windows and you'd rather ride through                          |
| `LW_GATEWAY_AUTH_CACHE_HARD_GRACE=0s` | **Restores legacy behaviour:** any refresh failure past JWT exp evicts immediately. Pick this if you operate under a security regime where stale-while-error is unacceptable | n/a (zero is the disable signal)                                                                                              |

**Alert pattern (log-based, no metric infra required):**

```bash theme={null}
# Fire if hard_evict reason=hard_cap_exceeded appears more than N times in 5 min
# Indicates the outage exceeded the configured grace and customers are now being rejected.
kubectl logs -n langwatch -l app=langwatch-gateway --since=5m | \
  grep -c 'auth_cache_hard_evict.*reason=hard_cap_exceeded'
```

A non-zero count is the page-worthy signal; the WARN spew alone (`auth_cache_refresh_transport_failure`) is informational while customers are still being served.

See [Config → Auth cache](/docs/ai-gateway/self-hosting/config#auth-cache) for the env-var contract.

## Graceful degradation: what survives what

The gateway is a cache of the control plane, so a surprising amount continues to work when pieces go down. Quick reference:

| Component down                | Customer impact                                     | How long gateway stays up                                                                                                                                | Notes                                                                                                                                                                                                                                                                  |
| ----------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LangWatch control plane       | None for cached VKs                                 | \~15 min JWT exp + `LW_GATEWAY_AUTH_CACHE_HARD_GRACE` (default 6 h) of stale-while-error; bootstrap cache + Redis L2 extend coverage to cold-for-pod VKs | Spend accounting is unaffected: cost rides the OTel span and the ledger catches up once the control plane returns. New VK creation is blocked until control plane returns. See [Recipe 8](#recipe-8--control-plane-outage--stale-while-error) for the operator runbook |
| One upstream provider         | None for VKs with fallback; 502/504 for VKs without | Indefinite, circuit breaker + fallback chain absorb it                                                                                                   | `gateway_circuit_state{credential_id}` shows which provider is down                                                                                                                                                                                                    |
| Redis L2 cache                | +\~30 ms on cold-for-pod VKs                        | Indefinite, L1 + `/resolve-key` cover the miss                                                                                                           | Fail-open by design (poison entries `DEL`, network errors log and miss)                                                                                                                                                                                                |
| Gateway pod (crash, eviction) | None, HPA replaces, PDB holds ≥ 2 replicas          | N/A                                                                                                                                                      | Stateless; new pod warms from Redis L2 + `/bootstrap`                                                                                                                                                                                                                  |

For multi-region deployments with shared control plane: a gateway region down is handled by Route53 latency-based failover. See [Scaling → Regional placement](/docs/ai-gateway/self-hosting/scaling#regional-placement).

## Writing findings back

When you find something worth filing:

1. Grab the `.pb.gz` with `go tool pprof -symbolize=remote -proto http://localhost:6060/debug/pprof/heap > /tmp/heap.pb.gz`.
2. Attach it to the issue along with:
   * Pod name + image digest (`kubectl get pod -n langwatch -o yaml | grep image`).
   * `X-LangWatch-Request-Id` from one exemplar bad request.
   * `kubectl top pod` output around the spike.
3. For urgent escalation, post in `#ai-gateway-support` with the request id.

## See also

* [Config → Admin, operator endpoints](/docs/ai-gateway/self-hosting/config#admin--operator-endpoints): env-var reference.
* [Health Checks](/docs/ai-gateway/self-hosting/health-checks): `/readyz` output to interpret alongside pprof.
* [Troubleshooting](/docs/ai-gateway/troubleshooting): symptom-first diagnostic index for non-pprof issues (401, 403, 429, 502).
* [Prometheus alerts](/docs/ai-gateway/cookbooks/prometheus-alerts): alert rules that tell you *when* to reach for this runbook.