Skip to main content
The gateway exposes three HTTP endpoints for Kubernetes probes plus one public status endpoint, all on the same port the API listens on (5563 by default, referenced as the named container port http in the chart). Each is deliberately scoped, /readyz flipping to 503 must mean “this replica should not serve customer traffic right now”, and nothing more.

​
Endpoint summary

The three probes are in-cluster signals. /health is the one meant to be reachable from outside; the chart publishes it through the ingress and leaves the rest internal. The three probes return the same JSON shape:
checks is omitted when there are no registered checks AND the status is ok. The gateway registers MarkStarted (at boot) and MarkDraining (on SIGTERM), but no per-dependency liveness or readiness checks. The probes are intentionally lightweight signals about process state and lifecycle, not external-dependency health. Dependency health is reported on /health, per-request error codes, and the OTel trace surface, never on the probes.

​
/healthz (liveness)

Cheap by design. Never does network I/O. If this returns non-200 the kubelet kills the pod. Chart default (charts/gateway/templates/deployment.yaml):

​
/readyz (readiness)

When the pod has received SIGTERM:
The loadbalancer drops a draining replica from rotation within seconds; in-flight requests on that replica continue to completion (graceful shutdown is described below). Do not weaken this probe: a replica flipping to draining must mean “stop sending new traffic here”, and the gateway only flips it on SIGTERM or via an explicit administrative call. Chart default:

​
What /readyz does NOT check

  • Per-provider live health. If OpenAI is rate-limiting, the gateway still serves and falls back per the VK’s configured chain. Reporting not_ready for one upstream would amplify the incident.
  • The control plane. A control-plane blip is a degradation the auth cache is built to absorb: warm keys keep serving from the in-process cache. Failing readiness on it would drop every replica from the load balancer at once and turn that degradation into a total outage. Control-plane reachability is reported on /health instead, where it informs a status page rather than gating traffic.
  • Redis L2. The current gateway has no Redis client; the auth cache is in-process LRU.
  • PostgreSQL. The gateway never talks to Postgres directly; the control plane mediates persistence.
The blast-radius of readiness is specifically: “is this pod still meant to serve traffic?”, and the only condition today is MarkDraining.

​
/startupz (startup)

There is no blocking bootstrap step today: MarkStarted is called while dependencies are being constructed, which finishes before the HTTP listener opens, so /startupz is already 200 on the first request it can possibly receive and the starting state is unobservable over HTTP. The auth cache warms organically on the first request per virtual key; an unwarmed gateway costs a cold-cache request one extra control-plane round trip, which is the correct behavior for a fresh deployment. The chart’s generous failure threshold is headroom kept for a future bootstrap-pull, not a window the gateway currently uses.
Once /startupz first returns 200, Kubernetes stops calling it; readiness + liveness take over.

​
/health (public status endpoint)

This is the endpoint to point a status page or uptime monitor at. Plain HTTP monitor semantics: 200 healthy, 503 not. HEAD returns the same status and headers with no body, as HTTP requires, so a monitor that probes with HEAD reads the same verdict. Responses carry Cache-Control: no-store so a CDN cannot serve a stale verdict. Any other method is 405: the ingress publishes the path, the gateway bounds the methods.
The body is deliberately smaller than the probes’: no version, no uptime_s. This endpoint is polled by the public internet, and restart cadence and build identity are not something to publish.

​
What it covers, and what it never will

The verdict covers the gateway process and the dependencies you own: the control plane. It is structurally independent of model providers.
  • A provider outage cannot turn it red. Nothing on the verdict path reads dispatch state, so if OpenAI and Anthropic are both down, completions fail while /health stays 200. That is the providers’ status page’s news, not yours, and a gateway that goes red on someone else’s outage trains people to ignore the page.
  • A poll never triggers an upstream call. A background monitor probes the control plane every 15 s and /health serves the cached verdict, so poll rate and control-plane load are unrelated. On an unauthenticated public endpoint, fanning out per poll would be an amplification and cost bug.

​
Control-plane semantics

The monitor probes the control plane’s /api/internal/gateway/health over the same HMAC-signed internal channel every request uses. That is the point of the design: a 200 proves DNS, TCP/TLS, the app being up, and the shared LW_GATEWAY_INTERNAL_SECRET matching. A mismatched secret is the misconfiguration where every pod looks green while every virtual-key resolve is refused, and this is the only signal that catches it. Unreachability shorter than 60 s stays 200: warm keys keep serving from the auth cache through a blip, and a page that flaps on what customers cannot feel is worse than no page. Past 60 s, cold-cache requests are failing with auth_upstream_unavailable and the page should say so. A booting pod is given one full 60 s window before it can report red, so a rolling deploy never blinks the page. Because the tolerance is 60 s and the probe interval is 15 s, a monitor polling every 30 to 60 s sees a sustained outage within about two minutes of onset.

​
Exposure

The chart publishes exactly this one path through the ingress, as an Exact match, so /healthz, /readyz, /startupz, /metrics and /internal stay in-cluster:
Point the monitor at https://gateway.your-corp.com/health. Set ingress.healthPath.enabled: false if you would rather not publish it at all; the endpoint still answers in-cluster.

​
Graceful shutdown

SIGTERM triggers (the chart’s shutdown.preDrainWait and shutdown.timeout knobs are not currently wired into the gateway code; the gateway uses Server.GracefulSeconds from pkg/config/server.go, default 5, bump via SERVER_GRACEFUL_SECONDS if you need a longer window):
  1. Immediately: MarkDraining() flips /readyz to 503 with status:"draining". The Service’s endpoint controller and the LB observe and stop routing new traffic.
  2. Drain window: existing in-flight requests continue. SSE streams continue until the upstream provider closes them or the request finishes naturally.
  3. End of GracefulSeconds: the HTTP server shuts down; remaining sockets close cleanly.
  4. Process exits 0.
Match your terminationGracePeriodSeconds to the drain window plus a few seconds of slack:
Without giving the LB time to notice /readyz=503 before the listener closes, a small fraction of in-flight requests during rolling deploys hits a replica that has already shut down, the LB returns 502. If you observe this on rollout, add a preStop sleep so the pod stays around long enough for the LB to remove it:

​
End-to-end synthetic check

Neither the probes nor /health exercise the request path, and /health will not tell you a provider is down, on purpose. For real-traffic confidence, run a synthetic completion every 30 s. Keep it on an internal alert rather than on the public status page, so a provider incident does not read as a LangWatch incident:
On alert:
  1. Capture the X-LangWatch-Request-Id header from the response.
  2. Look up the request’s trace in the LangWatch UI under Origin = gateway.
  3. If the gateway returned 5xx, pull the gateway pod logs filtered by pod name (GATEWAY_NODE_ID is unused; the gateway derives node_id from os.Hostname() which inside a Kubernetes pod is the pod name).
  4. If the gateway returned the upstream provider’s error verbatim, the gateway is healthy, the provider is degraded.

​
Common failures