Credential-leak incident response

The gateway gives you three operations on a leaked virtual key: disable, revoke and rotate. They differ in what happens to the old secret and in how fast the change reaches every pod. A leaked provider credential is a different problem. Rotate that one in the provider’s own console. See What not to do.

Pick the operation first

Operation	What happens to the old secret	Use it when
Disable	Every request answers `403 virtual_key_disabled`. An administrator can enable the key again, exactly as it was	You need traffic stopped now and you are unsure whether the key leaked
Revoke	Every request answers `403 virtual_key_revoked`. The key is dead and cannot come back	The secret is publicly exposed
Rotate	Keeps working for 24 hours, then fails	The secret left a trusted boundary but is not exposed, and callers need time to pick up the new one

Disable and revoke contain the leak as soon as the change reaches every gateway pod. The control plane writes a change event, and a pod that receives it stops serving the key on the next request, in about 10 seconds. A pod that cannot reach the control plane serves the cached key for longer: up to the token expiry of 15 minutes, and past it to the hard cap set by LW_GATEWAY_AUTH_CACHE_HARD_GRACE_SECONDS, six hours by default. Read Step 3 for the three paths and their timings, and report containment against the slowest one that applies. Rotate leaves the old secret working for a day, so it is a rollover tool rather than a containment tool. All three keep the key’s id, so its traces, its audit rows and its spend stay attributable.

Step 1: find the key

The first 13 characters of a secret are its display prefix, and the prefix is safe to paste into a ticket. Match it against the key list:

langwatch virtual-keys list --format json \
  | jq --arg p "vk-lw-01HZX9N" '.[] | select(.display_prefix == $p)'

In the app, open AI Gateway → Virtual Keys and match the prefix column. The row opens the key detail page.

Step 2: capture the state before you change anything

Rotation and revocation both write an audit row, but the questions you will be asked afterwards are about what the key was doing before you touched it.

Open the key detail page and read its Audit history, which is the audit log filtered to this key.

Read the key’s spend for the window you care about:

langwatch virtual-keys spend "$VK_ID" \
  --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z --format json \
  > incident-$VK_ID-before.json

Note the request pattern in Trace Explorer, filtered to this key.

Off-hours traffic or a spend jump that does not match your own callers is the signal that the leak is older than you think.

Step 3: stop the traffic

Revoke when the secret is public:

langwatch virtual-keys revoke "$VK_ID"

Or through the management API:

curl -sS -X POST \
  "https://app.langwatch.ai/api/gateway/v1/virtual-keys/$VK_ID/revoke" \
  -H "Authorization: Bearer $LANGWATCH_API_KEY"

Disable when you want the same stop and the option to undo it. Disable takes a reason, which lands in the audit row:

langwatch virtual-keys disable "$VK_ID" --reason "secret found in a public commit"

curl -sS -X POST \
  "https://app.langwatch.ai/api/gateway/v1/virtual-keys/$VK_ID/disable" \
  -H "Authorization: Bearer $LANGWATCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"reason":"secret found in a public commit"}'

Revocation is terminal, so a revoked key cannot be disabled and cannot be rotated. Disable first if you want the reason recorded, then revoke.

What happens to traffic already in flight

A request that has already been authenticated runs to completion. The three operations do not cancel it. New requests stop as each gateway pod learns about the change:

Path	Timing
The change feed, which carries the revoke or disable event and evicts the key from that pod’s cache	About 10 seconds
The key’s own token expiring, if the change feed is not delivering	15 minutes
The stale-serving window, if the pod cannot reach the control plane at all	Up to `LW_GATEWAY_AUTH_CACHE_HARD_GRACE_SECONDS` past the token expiry, default six hours

The third row is the case worth planning for. While the control plane is reachable, a revoked key is refused within seconds, because a 403 from the control plane evicts the cached entry at once with no grace. The grace window only covers a control plane the pod cannot reach, and during such an outage a key revoked just before it can keep serving to the cap. If you need every pod to forget the key immediately, restart the gateway Deployment. Every new pod starts with an empty cache and resolves each key from the control plane on its first request.

Step 4: rotate and roll the callers over

Rotation mints a new secret and keeps the old one valid for 24 hours, so callers can pick up the new value without a break.

langwatch virtual-keys rotate "$VK_ID" --format json | jq -r .secret \
  | vault kv put -mount=secret/gateway "$VK_ID" secret=-

curl -sS -X POST \
  "https://app.langwatch.ai/api/gateway/v1/virtual-keys/$VK_ID/rotate" \
  -H "Authorization: Bearer $LANGWATCH_API_KEY"

The response is the only time the new secret is readable. LangWatch stores a hash of it. Lose the response and you have to rotate again.

During the 24 hours both secrets resolve to the same key, so spend, budgets and traces stay on one id and you cannot tell the two apart from the request side. Push the new secret to every caller and confirm the rollout from your own configuration management, not from LangWatch. At 24 hours the old secret stops resolving and its callers get 401 invalid_api_key.

Step 5: read the audit trail

Every create, update, rotate, disable, enable and revoke is an audit row carrying the actor, the time, and the before and after state. Filter the audit log in the app to Target = virtual_key and the key’s id, then export it for the ticket. For a scripted export, read the log through the API. See Audit log for the row shape and the query. Three questions to answer from it:

When did the leak start? Look for a configuration change that widened the key, a scope added or a limit raised, next to a change in the spend pattern.
Who could act on the key? The actor column on every row, and the key’s permissions. See RBAC.
What did it reach? The key’s traces for the suspect window, and the spend for the same window.

What not to do

Do not delete the key. Revocation keeps the row, so the traces and the spend stay attributable. A deletion loses the record you need for the review.
Do not assume the provider credential is safe. If the same leak could have exposed the upstream API key, rotate that one in the provider’s console as well.
Do not rotate when the secret is public. Rotation leaves the old secret working for 24 hours. Revoke instead.
Do not paste the new secret into chat. Put it in the secret store first.

Get Started

SDK Integration

Virtual Keys & Budgets

Billing & Metering

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

Credential-leak incident response

Pick the operation first

Step 1: find the key

Step 2: capture the state before you change anything

Step 3: stop the traffic

What happens to traffic already in flight

Step 4: rotate and roll the callers over

Step 5: read the audit trail

What not to do

See also

​Pick the operation first

​Step 1: find the key

​Step 2: capture the state before you change anything

​Step 3: stop the traffic

​What happens to traffic already in flight

​Step 4: rotate and roll the callers over

​Step 5: read the audit trail

​What not to do

​See also

Pick the operation first

Step 1: find the key

Step 2: capture the state before you change anything

Step 3: stop the traffic

What happens to traffic already in flight

Step 4: rotate and roll the callers over

Step 5: read the audit trail

What not to do

See also