Skip to main content

​
Components

The LangWatch AI Gateway is two processes. The gateway service is a Go binary on the request path. The control plane is the LangWatch application, which owns the virtual keys, the routing policies, the budgets and the spend ledger.

​
Request flow

  1. The client sends Authorization: Bearer vk-lw-... to the gateway.
  2. The auth cache returns the key’s bundle, or the gateway calls POST /api/internal/gateway/resolve-key on the control plane and caches the answer.
  3. The pipeline runs the interceptors in order. A rejected request (rate limit, policy rule, budget, guardrail) returns before any provider is called, and still writes a spend admission record.
  4. The provider dispatch sends the request upstream and streams the response back.
  5. The spend interceptor writes a confirm or fail record to the spool. The drainer ships it to the control plane, which debits the ledger.

​
Control plane routes

The gateway signs every call to the control plane with HMAC-SHA256 over the shared secret LW_GATEWAY_INTERNAL_SECRET. An unsigned or incorrectly signed call is refused. The gateway reads the control plane location from LW_GATEWAY_BASE_URL. Every spend, budget and auth call depends on it, and a wrong value fails without an error on the request path, so set it explicitly on every deployment.

​
Change feed

Every mutation that the gateway must see appends a row to the change feed with a monotonic revision. The kinds are VK_CREATED, VK_CONFIG_UPDATED, VK_ROTATED, VK_REVOKED, VK_DISABLED, VK_ENABLED, BUDGET_CREATED, BUDGET_UPDATED, BUDGET_DELETED, MODEL_PROVIDER_UPDATED, ROUTING_POLICY_UPDATED, ROUTING_POLICY_DELETED, CACHE_RULE_CREATED, CACHE_RULE_UPDATED and CACHE_RULE_DELETED. The gateway long-polls GET /api/internal/gateway/changes?organization_id=...&since=<revision>&timeout_s=10 per organization it has served. The control plane holds the request open for up to timeout_s seconds (default 10, maximum 25), checks for new rows every 2 seconds, and returns up to 500 events with current_revision. With no events it returns 204 with the current revision in the X-LangWatch-Revision header, and the gateway polls again from there. On each event the gateway evicts or refreshes the affected cache entries. A revoked or disabled key stops serving on the next request after the event arrives.

​
Spend spool

The gateway never writes spend on the request path. The spend interceptor appends an admit record when a request enters the pipeline, and a confirm or fail record when it finishes, to an on-disk spool. The spool is at <os temp dir>/langwatch-gateway-spend-spool and is bounded to 64 MiB; when it is full, the oldest records are dropped and the gateway_spend_spool_dropped_total metric counts them. Records seal into segments every second. A drainer ships sealed segments oldest first to POST /api/internal/gateway/spend-commands, at least once: a segment is deleted only after the control plane acknowledges it. On a failed ship the drainer retries with a backoff from 1 second doubling to 60 seconds. On the control plane, the spend processing pipeline turns the commands into ledger rows in gateway_budget_ledger_events and appends a BUDGET_UPDATED change event for each budget it debited. The gateway then re-fetches the key config and sees the new spend on its next precheck.

​
Auth cache

The auth cache keeps each resolved key’s JWT and config in memory, keyed by a hash of the presented key. A hit costs no control plane call. The cache serves a stale entry when the control plane cannot be reached. After the JWT expires, a refresh that fails for a transport reason (network error, timeout, 5xx) extends the entry by the soft bump and keeps serving, up to the hard grace past the JWT expiry. A 401, 403 or 404 from the control plane evicts the entry at once. A key with its own expiration date is refused past that date with virtual_key_expired without a control plane call. The config part of the entry (credentials, routing chain, budgets) has its own TTL. Past it, the gateway re-fetches the config in the background with the stored ETag, so a config that has not changed costs a 304.

​
Budget enforcement

The budget precheck runs on the gateway against the spend figures in the cached config, so the check costs no control plane call. A request over a limit is refused before dispatch. A per-end-user budget reads the bucket’s spend from /api/internal/gateway/budget-bucket-spend through a cache, and a request with such a budget and no end-user id is refused. The check is permissive on error: when the spend figure cannot be read, the request is allowed and the debit reconciles later. A budget filtered to one provider does not block the request; it removes that provider from the routing chain, and only an emptied chain returns a budget error.

​
Deployment

The gateway sub-chart of the langwatch Helm chart deploys the gateway as a Deployment with a Service, an optional Ingress, an HPA, a PodDisruptionBudget and a NetworkPolicy. The spool lives on the pod’s writable emptyDir, so a pod that is killed loses at most the records still in the spool, about a second of drain lag. Also check: Helm deployment for the chart values, and Budgets for the budget scopes and periods.
Last modified on September 6, 2026