Helm chart

Documentation Index

Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

Pre-create the runtime secrets

# Default Secret name when installing the umbrella `langwatch` chart.
# (Gateway sub-chart standalone uses `gateway-runtime-secrets` instead;
# pass --set gateway.secrets.existingSecretName=... to override either.)
kubectl create secret generic langwatch-gateway-auth \
  --namespace langwatch \
  --from-literal=LW_GATEWAY_INTERNAL_SECRET="$(openssl rand -hex 32)" \
  --from-literal=LW_GATEWAY_JWT_SECRET="$(openssl rand -hex 32)"

​

Image registry and tag

gateway:
  image:
    repository: docker.io/langwatch/ai-gateway   # default
    tag: ""                                       # default → .Chart.AppVersion (lockstep)
    pullPolicy: IfNotPresent

​

Enabling the gateway

gateway:
  chartManaged: true                              # default; flip to false to disable

  # Where the gateway reaches the control plane. The default targets the
  # langwatch-app Service in the same namespace.
  controlPlane:
    baseUrl: http://langwatch-app:5560

  # Secret you pre-created above.
  secrets:
    existingSecretName: langwatch-gateway-auth    # default for the umbrella chart
    internalSecretKey: LW_GATEWAY_INTERNAL_SECRET # key WITHIN that Secret
    jwtSecretKey: LW_GATEWAY_JWT_SECRET           # key WITHIN that Secret

  # OTLP endpoint the gateway exports spans to. Default targets the
  # control plane's /api/otel ingest, which routes spans into the
  # owning project via the langwatch.project_id span attribute.
  otel:
    endpoint: http://langwatch-app:5560/api/otel
    # Production: provide the X-Auth-Token via a pre-existing Secret.
    authExistingSecretName: ""
    authSecretKey: token

  ingress:
    enabled: true
    className: nginx
    host: gateway.your-corp.com
    path: /v1                                     # only /v1/* needs to be public
    pathType: Prefix
    tls:
      enabled: true
      secretName: gateway-tls

  # Edge protection. Cap request body before auth runs so a misconfigured
  # caller can't OOM the pod. 10 MiB fits ~3 MiB prompts plus base64 images.
  security:
    maxRequestBodyBytes: 10485760                 # 10 MiB

helm upgrade --install langwatch langwatch/langwatch \
  -f values.yaml \
  --namespace langwatch --create-namespace

​

Env vars wired by the chart

​

Secret rotation

gateway:
  secrets:
    existingSecretName: langwatch-gateway-auth
    jwtSecretKey: LW_GATEWAY_JWT_SECRET
    # Rotation window only: which key in the same Secret holds the retired
    # value. Empty = strict single-key mode (production steady state).
    jwtSecretPreviousKey: LW_GATEWAY_JWT_SECRET_PREVIOUS

1. Flip the control plane’s signing secret to the new value (rotate LW_GATEWAY_JWT_SECRET on the langwatch-app Deployment first — the issuer must accept the new value before the gateway starts seeing it on inbound JWTs).
2. Update the langwatch-gateway-auth Secret: keep LW_GATEWAY_JWT_SECRET pointing at the new value, add LW_GATEWAY_JWT_SECRET_PREVIOUS carrying the old value. helm upgrade with gateway.secrets.jwtSecretPreviousKey: LW_GATEWAY_JWT_SECRET_PREVIOUS.
3. Rolling restart picks up both keys. Tokens signed under the old key verify against LW_GATEWAY_JWT_SECRET_PREVIOUS; new tokens use LW_GATEWAY_JWT_SECRET.
4. After ~15 min (longest pre-rotation bundle TTL), remove LW_GATEWAY_JWT_SECRET_PREVIOUS from the Secret and unset jwtSecretPreviousKey. Rolling restart. Strict mode resumes.

kubectl logs -n langwatch deploy/langwatch-gateway | grep jwt_secret_rotation_active

​

Dependencies on the control plane

• langwatch-app on gateway.controlPlane.baseUrl (default http://langwatch-app:5560). The gateway hits /api/internal/gateway/resolve-key, /changes, and /budget/check synchronously. If unreachable, the gateway extends its cached entries via stale-while-error (see LW_GATEWAY_AUTH_CACHE_SOFT_BUMP / HARD_GRACE) but new VKs cannot resolve and rate limits cannot tighten.
• langwatch-workers running the trace-processing pipeline. Budget enforcement depends on this. The gateway emits OTLP spans with cost attrs but does NOT debit budgets directly — the workers’ trace-fold reactor (gatewayBudgetSync.reactor.ts) reads finalised spans, computes cost, and inserts ledger rows into ClickHouse. The gateway_budget_scope_totals_mv materialized view aggregates spend, and /budget/check reads from that view. If workers.enabled: false, gateway traces still flow but spend never accumulates against budgets, so hard-cap scopes will not block at the configured limit. Self-hosters running the umbrella chart get workers by default; standalone charts/gateway installs MUST point at a control plane that has the worker pipeline running.

​

Networking

• The LangWatch control plane (gateway.controlPlane.baseUrl → LW_GATEWAY_BASE_URL) for /api/internal/gateway/* calls.
• Outbound to each provider API (OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex, Gemini). NetworkPolicy must allow egress (see below).

​

Scaling

​

High availability

• Run ≥2 replicas with a PodDisruptionBudget (chart default minAvailable: 1) so rolling updates don’t remove every pod simultaneously.
• The auth-cache stale-while-error grace window (LW_GATEWAY_AUTH_CACHE_SOFT_BUMP / HARD_GRACE) keeps replicas serving with cached VK bundles when the control plane is briefly offline. See Auth-cache stale-while-error for the soft/hard-grace knobs.
• The Redis L2 cache (gateway.redis.url) is forward-compat surface today — the v1 gateway has no Redis client, so this knob is a no-op. v1.1 will wire it for replica-warm-up on HPA scale-out. Don’t rely on it for cold-start coverage in v1; the auth-cache grace window is the only operational backstop.

​

Monitoring

​

Probes

​

Startup network check

gateway:
  startup:
    # Leave empty to disable (default, backwards-compatible).
    # Comma-separated host:port list (same wire format as GATEWAY_STARTUP_NETCHECK_HOSTS).
    netcheckHosts:
      - api.openai.com:443
      - api.anthropic.com:443
      - generativelanguage.googleapis.com:443
    netcheckTimeout: 2s

# Grep the pod logs after a failed rollout
kubectl logs -n langwatch deploy/langwatch-gateway | grep -E 'dns resolution failed|tcp dial failed'

​

Admin listener

​

Posture 1: disabled

gateway:
  admin:
    addr: ""

​

Posture 2: loopback-only (default)

gateway:
  admin:
    addr: "127.0.0.1:6060"
    # existingAuthSecretName: <unset>   # optional, works as a defence-in-depth layer

​

Posture 3: bearer-gated non-loopback

gateway:
  admin:
    addr: "0.0.0.0:6060"
    existingAuthSecretName: gateway-admin-token
    authSecretKey: token                # key within the Secret; default "token"

kubectl create secret generic gateway-admin-token \
  --from-literal=token="$(openssl rand -hex 32)" \
  -n langwatch

• non-k8s deploys (systemd, Nomad, plain Docker) where kubectl port-forward isn’t available;
• k8s deploys behind a corporate VPN or internal LB where operators want direct HTTP access without going through kubectl exec.

​

Safety net (v1.1)

Non-loopback GATEWAY_ADMIN_ADDR + empty GATEWAY_ADMIN_AUTH_TOKEN + unset GATEWAY_ALLOW_INSECURE ⇒ the gateway will refuse to start with an error naming both env vars.

​

Accessing pprof with the bearer token

TOKEN=$(kubectl get secret gateway-admin-token -n langwatch -o jsonpath='{.data.token}' | base64 -d)

# pprof's -source_token flag threads the Authorization header through
go tool pprof \
  -source_token "Bearer $TOKEN" \
  http://gateway.your-corp.internal:6060/debug/pprof/heap

# Or curl-based workflow
curl -sS -H "Authorization: Bearer $TOKEN" \
  http://gateway.your-corp.internal:6060/debug/pprof/goroutine?debug=1 \
  | head -100

​

Observability

{"msg":"admin_listening","addr":"0.0.0.0:6060","auth_required":true,"loopback_only":false}

​

NetworkPolicy

gateway:
  networkPolicy:
    enabled: true

    # Optional — restrict OTLP egress to a specific namespace/pod.
    # Leave empty to skip the OTLP egress rule entirely (not exported externally).
    egressToOTLP:
      - namespaceSelector: { matchLabels: { name: observability } }
        podSelector: { matchLabels: { app: otel-collector } }

    # Optional — lock provider egress to explicit CIDRs (stricter than default).
    # Default: allow any IP EXCEPT RFC1918 ranges (see below).
    egressToProviders:
      - cidr: 104.18.0.0/16    # cloudflare (fronts most providers)
      - cidr: 52.0.0.0/8       # aws us-east-1 (Anthropic, Bedrock)

​

What the default policy allows

• ingress-nginx namespace → port 5590 (customer traffic via your Ingress).
• monitoring namespace Prometheus pod → port 5590 /metrics.

• DNS to kube-system on TCP+UDP :53 (ordering matters — DNS rule MUST come first or other rules fail to resolve hostnames).
• Control plane (langwatch-app label selector) on port 5560 for /resolve-key / /config / /changes / /budget / /guardrail. Matches the langwatch chart’s langwatch-app Service (app=5560, nlp=5561, langevals=5562, gateway=5563).
• Redis on :6379 — only rendered when gateway.redis.url is set.
• OTLP :4318 — only rendered when gateway.networkPolicy.egressToOTLP is non-empty.
• Provider upstreams on :443 — default any IP EXCEPT RFC1918 (10/8, 172.16/12, 192.168/16). Teams with explicit compliance posture should override egressToProviders with a concrete CIDR allowlist.

​

Verifying the policy rendered

helm template langwatch langwatch/langwatch \
  -f values.yaml \
  --set gateway.networkPolicy.enabled=true \
  | grep -A2 'kind: NetworkPolicy'

​

Inside the pod

# Ingress check — should succeed from ingress-nginx, fail from elsewhere
kubectl exec -n langwatch deploy/langwatch-gateway -- curl -sS http://localhost:5590/healthz

# Egress check — should succeed to api.openai.com, fail to a lateral pod
kubectl exec -n langwatch deploy/langwatch-gateway -- \
  curl -sS --max-time 3 https://api.openai.com/v1/models

​

Graceful drain

1. SIGTERM received. ctx.Done() fires.
2. Flip readiness + gauge. /readyz starts returning 503 {"status":"draining"}; gateway_draining{pod=...} goes to 1.
3. Pre-drain wait (shutdown.preDrainWait, default 5 s). Gives the LB endpoint controller (nginx-ingress + kube-proxy) time to remove the pod from service endpoints. EKS-observed propagation is 3-4 s; 5 s has margin. During this window the pod is refusing new work via /readyz but /livez + /startupz stay green so kubelet doesn’t short-circuit the drain.
4. Graceful close. server.Shutdown(Timeout=15s) blocks until every in-flight handler exits, then force-closes. Streaming handlers that honour request context cancellation finish cleanly within the grace.

gateway:
  shutdown:
    preDrainWait: 5s
    timeout: 15s
  terminationGracePeriodSeconds: 30   # MUST be ≥ preDrainWait + timeout + slack

​

Invariant

gateway:
  shutdown:
    preDrainWait: 15s
    timeout: 20s
  terminationGracePeriodSeconds: 45   # 15 + 20 + 10 slack

​

Observability

• gateway_draining{pod} — 0 normally, 1 during shutdown. A stuck pod (draining but never dying) shows as this being 1 for > grace.
• gateway_in_flight_requests{pod} — monotonic counter minus completions, so concretely the count of currently-executing handlers. During drain this should curve down to 0.

​

Security

gateway:
  security:
    maxRequestBodyBytes: 10485760   # 10 MiB, default

• security — edge, fails fast with 413, operator bumps when large base64 images are legitimate;
• budget — accounting, fails async via outbox, operator investigates debit flow (see Recipe 6);
• upstream — per-provider, fails mid-flight with circuit-breaker trips.

​

When to tune

• Lower to ~1 MiB if you’re fronting a tightly-scoped internal API with known small payloads and want to reject bot scans at the cheapest possible cost.
• Raise to ~50 MiB when a customer’s workload includes large base64-encoded images in vision messages. You’ll know you need it when legitimate requests start returning 413 payload_too_large; the troubleshooting entry in Troubleshooting tells users exactly which env var to bump.
• Do not disable. A body cap is the single cheapest defence against drive-by memory pressure on a public endpoint.

​

HTTP server timeouts

gateway:
  security:
    readHeaderTimeout: 5s    # TLS-handshake-then-stall probes
    readTimeout: 60s         # full body read; stops byte-trickle
    idleTimeout: 120s        # keep-alive cap; MUST exceed nginx keepalive_timeout

• writeTimeout is NOT exposed. It would bound the whole response, which terminates long SSE streams (reasoning models, Claude thinking traces) at the cap. Per-chunk streaming deadlines live in the dispatcher, not here. If you’re behind an ingress that enforces its own proxy_read_timeout, match it there instead.
• idleTimeout MUST exceed your nginx-ingress keepalive_timeout (typical 75 s). If the gateway closes first, nginx’s pool has dead sockets and a request fails mid-flight. The chart default 120 s is safe under the typical 75 s nginx value; if you run a custom ingress with a longer keepalive, bump this value proportionally.

​

Chart and data-plane CI

• Go data plane: go mod verify, go vet, go build, go test -count=1 -race ./... across all 14 internal packages. Benchmarks compile under -run=^$ -benchtime=1x so Go API drift fails fast without adding benchmark noise to PR logs.
• Helm chart: helm lint + two helm template renders — one with defaults (must render zero NetworkPolicy objects), one with networkPolicy.enabled=true + redis.url=redis://… (must render exactly one). Both invariants are asserted explicitly; violating either fails the job.

​

Upgrade procedure

1. helm diff upgrade langwatch langwatch/langwatch -f values.yaml --version <new> to preview the change set.
2. helm upgrade langwatch langwatch/langwatch -f values.yaml --version <new>. The umbrella’s langwatch-app and gateway Deployments roll concurrently — both use Kubernetes’ standard RollingUpdate strategy and the chart does not enforce ordering. Safety during the overlap window is handled by the dual-key signature contract documented under Secret rotation: the new control-plane image accepts both old and new signatures, so a brief window where gateway pods on the new image talk to control-plane pods still on the old image (or vice versa) does not break authentication. If you want to enforce ordering for an extra-cautious rollout, run helm upgrade ... --set gateway.enabled=false first to roll the app alone, then re-enable the gateway in a second helm upgrade — but for the standard upgrade path the dual-key contract is what makes the concurrent roll safe.
3. Monitor gateway_provider_duration_seconds and gateway_requests_total{status!="2xx"} for ~30 min.
4. Roll back via helm rollback langwatch <revision> if anomalies appear — the gateway’s cache survives restarts and requests resume immediately. Both sub-charts roll back together.