How a request gets its cache action
The gateway decides one cache action for every request, in this order:- Cache rules. If the organization has cache rules, the gateway evaluates them against the request and the virtual key. The first rule that matches sets the action.
- Provider default. If no rule matches and the request resolves to an Anthropic model, the gateway applies
forceon its own. This default only runs when the body is 4096 bytes or larger and the client is not already managing its own breakpoints: on/v1/messagesanycache_controlmarker in the body disables it, on/v1/chat/completionsonly a top-levelcache_controldoes. - Passthrough. Otherwise the body reaches the provider unchanged.
X-LangWatch-Cache-Mode reports the outcome: respect, disable or force when a rule fired, auto when the provider default fired. The header is absent when neither a rule nor the provider default changed the body.
Actions
There is no time-to-live action. The TTL and salt fields on a rule are stored but the gateway does not read them.
What force injects
On POST /v1/messages (Anthropic shape), the gateway sets cache_control: {"type": "ephemeral"} on the last system block and on the last content block of the last message. Any cache_control the client already set is left in place.
On POST /v1/chat/completions (OpenAI shape), the gateway sets a top-level cache_control: {"type": "ephemeral"} on the body. For Anthropic models this becomes the Messages API request-level cache_control, which places a breakpoint after the last cacheable block, so tools, system prompt and the conversation so far are reused turn over turn. Providers that cache automatically (OpenAI, Azure OpenAI, Gemini) ignore the field.
On every other endpoint force leaves the body unchanged.
Cache rules
A cache rule is an organization-wide match plus an action. Rules apply to every virtual key in the organization; narrow a rule with its matchers. Manage rules under AI Gateway, Cache Rules in the sidebar (/gateway/cache-rules).
The Cache Rules list. The rule with the lower priority number is checked first.
Matchers
All matchers you set must match for the rule to fire. A matcher you leave empty matches everything.
The Match when block of the New cache rule drawer. Every field you fill must match. With only the model pattern filled, the rule matches every key.
claude-* matches claude-haiku-4-5 and does not match anthropic/claude-haiku-4-5. Write the pattern for the names your callers send, or add a second pattern with the prefix.
Leave the principal field empty. The gateway does not pass a principal to the rule evaluation, so a rule that sets one never fires. The request metadata key and value are not evaluated: a rule that sets them matches as if they were empty.
Action
The Then do block sets one of the three actions.
The Then do block with the action set to force.
Priority
Priority is an integer from 0 to 1000, default 100. The gateway evaluates rules in ascending priority number: the rule with the lowest number is checked first, and the first rule that matches wins.Propagation
Creating, editing or archiving a rule writes a change event in the same transaction. The gateway picks the event up on its next change-feed poll and rebuilds the organization’s key bundles, so the next request runs against the new rules. No restart is needed. Archiving a rule stops it from matching. The audit log keeps the rule’s history.Verify a rule from the response
Send a request through a key the rule matches and read the response headers:gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens.
On the response side, Anthropic reports cache_read_input_tokens and cache_creation_input_tokens in usage. The gateway forwards these counts unchanged and records them on the span and in the spend record, so cached and uncached input tokens are priced separately.
Rollout recipes
Force caching for one team
- Tag the team’s virtual keys, for example
team=support. - Create a rule: priority
50, virtual key tagsteam=support, modelclaude-*, actionforce. - Send one request through a tagged key and check
X-LangWatch-Cache-Mode: force. - Compare
cache_read_input_tokenson the second identical request against the first.
Disable caching for evaluation traffic
Evaluation runs should not reuse a cached prefix, so that every run pays the same cost and sees the same behavior.- Give the evaluation keys a tag, for example
suite=evals. - Create a rule: priority
10, virtual key tagssuite=evals, actiondisable. - Verify with
X-LangWatch-Cache-Mode: disableon a request through one of those keys. Everycache_controlblock the client set is gone from the body the provider receives.
Roll back
Archive the rule from the row menu on the Cache Rules page. The next change-feed poll removes it from every key bundle. The rule’s create, edit and archive events stay in the audit log.Boundaries
- A cache mode set on a virtual key’s own configuration is not applied. The gateway reads cache rules and the provider default only.
- Rules change the request body only. The gateway has no cache of its own; the provider decides whether a cached prefix is served.
- OpenAI, Azure OpenAI and Gemini cache automatically on the provider side.
forceanddisabledo not change their behavior;X-LangWatch-Cache-Modestill reports the action that fired. - The provider default (
auto) never overrides a client that manages its own breakpoints. On/v1/messagesanycache_controlin the body disables it; on/v1/chat/completionsonly a top-levelcache_controldoes, because message-level markers do not survive the translation to the Anthropic shape.