Helm chart

The gateway is a sub-chart of the langwatch Helm chart. The umbrella chart enables it by default with gateway.chartManaged: true and runs it as its own Deployment, Service and Ingress next to langwatch-app. This guide takes you from an empty namespace to a gateway that answers a chat completion.

Prerequisites

A Kubernetes cluster and helm 3.
An ingress controller. The chart defaults to ingress.className: nginx.
A hostname for the gateway, separate from the app hostname. See DNS and TLS.
The LangWatch umbrella chart. Follow Kubernetes (Helm) for the app side first if you have not installed it yet.

Create the app Secret

The gateway pod and the app pod mount the same Secret. The gateway reads two keys from it: LW_GATEWAY_INTERNAL_SECRET signs every call between the gateway and the control plane, and LW_GATEWAY_JWT_SECRET verifies the JWTs the control plane issues for resolved virtual keys.With autogen.enabled=true the chart creates the Secret on first install. Otherwise create it before helm install:

kubectl create namespace langwatch

kubectl -n langwatch create secret generic langwatch-app-secrets \
  --from-literal=credentialsEncryptionKey="$(openssl rand -hex 32)" \
  --from-literal=cronApiKey="$(openssl rand -hex 32)" \
  --from-literal=nextAuthSecret="$(openssl rand -hex 32)" \
  --from-literal=virtualKeyPepper="$(openssl rand -hex 32)" \
  --from-literal=LW_GATEWAY_INTERNAL_SECRET="$(openssl rand -hex 32)" \
  --from-literal=LW_GATEWAY_JWT_SECRET="$(openssl rand -hex 32)"

gateway.secrets.existingSecretName (default langwatch-app-secrets) must equal the app Secret name. If you set secrets.existingSecret or autogen.secretNames.app to another name, set gateway.secrets.existingSecretName to the same name. The chart refuses to render when the two differ.virtualKeyPepper is read by the app only. The gateway never mounts it.

Write the gateway values

Add a gateway block to your umbrella values.yaml:

gateway:
  chartManaged: true

  # Externally reachable URL the control plane hands to CLI users at login.
  # Leave empty to derive https://<ingress.host> from the ingress below.
  publicUrl: ""

  # Where the gateway reaches the control plane. Empty resolves to
  # http://<release>-app:5560, the app Service of this install.
  controlPlane:
    baseUrl: ""

  secrets:
    existingSecretName: langwatch-app-secrets
    internalSecretKey: LW_GATEWAY_INTERNAL_SECRET
    jwtSecretKey: LW_GATEWAY_JWT_SECRET
    jwtSecretPreviousKey: ""      # set only during a JWT secret rotation

  ingress:
    enabled: true
    className: nginx
    host: gateway.acme.com        # your hostname, never the chart placeholder
    path: /v1
    pathType: Prefix
    healthPath:
      enabled: true               # publishes GET /health for uptime monitors
      path: /health
    tls:
      enabled: true
      secretName: gateway-tls
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/proxy-buffering: "off"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

  security:
    maxRequestBodyBytes: 33554432 # 32 MiB, applied before auth
    blockLocalHTTPCalls: false
    requireHTTPSCustomEndpoints: false
    allowedProxyHosts: []

  # OTLP/HTTP endpoint for the gateway's own spans. Empty disables export.
  otel:
    endpoint: ""

  spend:
    enabled: true                 # off means budgets are never debited

  shutdown:
    preDrainWaitSeconds: 5
    timeoutSeconds: 60
  terminationGracePeriodSeconds: 75

The image is docker.io/langwatch/ai-gateway. image.tag defaults to the chart appVersion. Set image.repository to your mirror if the cluster cannot pull from Docker Hub.

Install or upgrade

helm repo add langwatch https://langwatch.github.io/langwatch
helm repo update

helm upgrade --install langwatch langwatch/langwatch \
  --namespace langwatch --create-namespace \
  -f values.yaml \
  --wait --timeout 10m

The umbrella chart bundles the gateway sub-chart, so one helm upgrade rolls the app and the gateway together. The gateway Deployment uses a RollingUpdate with maxSurge: 1 and maxUnavailable: 0.

Verify

kubectl -n langwatch rollout status deploy/langwatch-gateway
kubectl -n langwatch run curl --rm -it --image=curlimages/curl --restart=Never \
  -- curl -sS http://langwatch-gateway/health

Expected output:

{"status":"ok","checks":{"gateway":"ok","control_plane":"ok"}}

Then run the post-install checklist to send a first completion through the gateway.

What the chart renders into the pod

The chart passes these environment variables to the container. Every other key in the gateway values.yaml (cache.*, bifrost.*, guardrails.*, budget.*, startup.*, otel.auth*, otel.batchTimeout, otel.maxQueueSize, security.*Timeout, controlPlane.*Timeout) does not reach the pod.

Env var	Value source	Purpose
`SERVER_ADDR`	fixed `:5563`	Listen address. The Service targets port 5563
`LOG_LEVEL`	`logging.level`	`debug`, `info`, `warn` or `error`
`LW_GATEWAY_BASE_URL`	`controlPlane.baseUrl`, else `http://<release>-app:5560`	Control plane URL
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_OTLP_ENDPOINT`	`otel.endpoint`, only when set	OTLP/HTTP endpoint for the gateway’s own spans
`LW_GATEWAY_SPEND_ENABLED`	`spend.enabled`	Spend record emission
`SERVER_MAX_REQUEST_BODY_BYTES`	`security.maxRequestBodyBytes`	Request body cap
`SERVER_DRAIN_DELAY_SECONDS`	`shutdown.preDrainWaitSeconds`	Wait after `/readyz` flips to 503
`SERVER_GRACEFUL_SECONDS`	`shutdown.timeoutSeconds`	In-flight request budget on shutdown
`BLOCK_LOCAL_HTTP_CALLS`	`security.blockLocalHTTPCalls`	Refuse provider endpoints on private addresses
`REQUIRE_HTTPS_CUSTOM_ENDPOINTS`	`security.requireHTTPSCustomEndpoints`	Refuse `http://` provider endpoints
`ALLOWED_PROXY_HOSTS`	`security.allowedProxyHosts`, comma-joined	Private hostnames exempt from the block
`LW_GATEWAY_INTERNAL_SECRET`	Secret `secrets.existingSecretName`, key `secrets.internalSecretKey`	HMAC secret for control plane calls
`LW_GATEWAY_JWT_SECRET`	Secret `secrets.existingSecretName`, key `secrets.jwtSecretKey`	JWT verification secret
`LW_GATEWAY_JWT_SECRET_PREVIOUS`	Secret, key `secrets.jwtSecretPreviousKey`, only when set	Previous JWT secret during rotation

The chart renders no authentication header for the OTLP export. Point otel.endpoint at a collector that accepts unauthenticated OTLP/HTTP, or leave it empty. The full variable reference is on Config. Write maxRequestBodyBytes as an integer literal. The chart renders it with %d, and a YAML float such as 3.3554432e+07 fails the render.

Rotate the JWT secret without downtime

The gateway verifies a JWT against LW_GATEWAY_JWT_SECRET first, then against LW_GATEWAY_JWT_SECRET_PREVIOUS when that variable is set. The control plane signs with the LW_GATEWAY_JWT_SECRET key of the same Secret, and a JWT lives 15 minutes.

Edit the Secret: copy the current LW_GATEWAY_JWT_SECRET value into a new key LW_GATEWAY_JWT_SECRET_PREVIOUS, then replace LW_GATEWAY_JWT_SECRET with the new value.
Set gateway.secrets.jwtSecretPreviousKey: LW_GATEWAY_JWT_SECRET_PREVIOUS and run helm upgrade. The gateway pods restart with both values. Wait for kubectl -n langwatch rollout status deploy/langwatch-gateway.
Run kubectl -n langwatch rollout restart deploy/langwatch-app. New JWTs are signed with the new value; JWTs signed before the restart still verify against the previous value.
After 15 minutes, delete the LW_GATEWAY_JWT_SECRET_PREVIOUS key from the Secret, clear jwtSecretPreviousKey, and run helm upgrade again.

LW_GATEWAY_INTERNAL_SECRET has no previous-key slot. Change it in the Secret and restart the app and the gateway Deployments together.

Control plane dependency

The gateway calls these routes on LW_GATEWAY_BASE_URL, signed with LW_GATEWAY_INTERNAL_SECRET: /api/internal/gateway/resolve-key, /config, /changes, /health, /budget-bucket-spend and /realtime-sessions. It also ships spend records to the control plane’s spend ingest. When the control plane is unreachable, virtual keys already in the auth cache keep serving under the stale-while-error rules on Config. Keys not in the cache fail with 503 auth_upstream_unavailable. Budget debits come from the spend records the gateway ships. With spend.enabled: false the gateway serves traffic and debits no budget. The spool lives in the pod’s /tmp emptyDir (scratchSizeLimit, default 1Gi) and is capped at 64 MiB. Alert on gateway_spend_spool_dropped_total: every drop is spend that is never debited.

Graceful drain

On SIGTERM the gateway:

Flips /readyz to 503 {"status":"draining"} and sets the gateway_draining gauge to 1.
Waits shutdown.preDrainWaitSeconds (default 5) so the endpoint controller removes the pod from the Service.
Stops the HTTP listener and gives in-flight requests up to shutdown.timeoutSeconds (default 60) to finish, then stops the other services in reverse start order. The spend spool closes after the listener, so requests that finish during the drain still record spend.

terminationGracePeriodSeconds must be at least preDrainWaitSeconds + timeoutSeconds + 10. The chart refuses to render otherwise. It also refuses the retired keys shutdown.preDrainWait and shutdown.timeout. A streaming response can run up to 14 minutes, the provider request timeout. With the default timeoutSeconds: 60 the gateway logs graceful_shutdown_shorter_than_max_stream_duration at boot, and a stream still open after 60 seconds of drain is cut. To finish long streams on every rollout:

gateway:
  shutdown:
    preDrainWaitSeconds: 5
    timeoutSeconds: 840
  terminationGracePeriodSeconds: 855

Watch gateway_draining together with gateway_in_flight_requests. During a healthy drain the in-flight count falls to 0. A flat count past the window is a stuck handler, and the kubelet kills the pod when the grace period ends.

NetworkPolicy

networkPolicy.enabled is false by default. When enabled the chart renders one deny-by-default policy:

Direction	Allowed	Value
Ingress	Pods in the `ingress-nginx` namespace, and Prometheus pods in the `monitoring` namespace, to port 5563	`networkPolicy.ingressFrom`
Egress	DNS to `kube-system` on TCP and UDP 53	fixed
Egress	Pods labelled `app.kubernetes.io/name: langwatch-app` on port 5560	`networkPolicy.egressToControlPlane`, `networkPolicy.controlPlanePort`
Egress	OTLP collector on port 4318, only when the list is non-empty	`networkPolicy.egressToOTLP`
Egress	Any public IP on TCP 443, with private, loopback, link-local and cloud metadata ranges excluded	`networkPolicy.egressToProviders`, `networkPolicy.providerPorts`

Replace egressToProviders with explicit CIDR blocks when your policy requires a pinned provider egress set. Check the namespace labels on your cluster before enabling: the defaults match upstream ingress-nginx and kube-prometheus-stack. Confirm the policy renders:

helm template langwatch langwatch/langwatch -f values.yaml \
  --set gateway.networkPolicy.enabled=true | grep -A2 'kind: NetworkPolicy'

Pod security

The pod runs as user and group 65532 with runAsNonRoot, a read-only root filesystem, all capabilities dropped, the RuntimeDefault seccomp profile and automountServiceAccountToken: false. The only writable path is the /tmp emptyDir. Adjust podSecurityContext and containerSecurityContext only if an admission policy on your cluster requires it.

Roll back

helm history langwatch -n langwatch
helm rollback langwatch <revision> -n langwatch

Every new gateway pod starts with an empty auth cache and resolves each virtual key from the control plane on its first request.

Common failures

Symptom	Cause	Fix
Pod in `CreateContainerConfigError`	The Secret named by `gateway.secrets.existingSecretName` or one of its keys does not exist	Create the Secret from step 1, then delete the pod
`helm upgrade` fails naming `gateway.secrets.existingSecretName`	The gateway Secret name differs from the app Secret name	Set both to the same name
`helm upgrade` fails naming `terminationGracePeriodSeconds`	Drain timing exceeds the grace period	Raise `terminationGracePeriodSeconds` to the number in the message
Pods healthy, every `/v1` call answers `503 auth_upstream_unavailable`	`LW_GATEWAY_BASE_URL` points at the wrong host or port	Set `gateway.controlPlane.baseUrl` to the app Service URL
`/health` answers 503 with `control_plane: unreachable for Ns`	Control plane unreachable, or `LW_GATEWAY_INTERNAL_SECRET` differs between the two pods	Compare the value in both Deployments, then check the Service and DNS
Pod refuses to start with `LW_GATEWAY_AUTH_CACHE_... is no longer read`	A retired duration-string variable is still set	Set the `_SECONDS` variable named in the message instead

More symptoms are on Troubleshooting.

Get Started

SDK Integration

Virtual Keys & Budgets

Billing & Metering

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

Prerequisites

What the chart renders into the pod

Rotate the JWT secret without downtime

Control plane dependency

Graceful drain

NetworkPolicy

Pod security

Roll back

Common failures

​Prerequisites

​What the chart renders into the pod

​Rotate the JWT secret without downtime

​Control plane dependency

​Graceful drain

​NetworkPolicy

​Pod security

​Roll back

​Common failures

Prerequisites

What the chart renders into the pod

Rotate the JWT secret without downtime

Control plane dependency

Graceful drain

NetworkPolicy

Pod security

Roll back

Common failures