Cache Control

​

The control surface: four layers (v1)

1. Gateway default: respect (byte-for-byte passthrough). Always safe, never surprises.
2. Per-VK default: respect, disable, force configurable in the VK drawer under “Cache control”. ttl=NNN is deferred to v1.1 at the per-VK level.
3. Cache rules: simple match/action overrides evaluated at auth-resolve time (bundled with VK config → near-zero hot-path cost). See Cache rules.
4. Per-request header: X-LangWatch-Cache: respect|disable|force on an individual request. Highest precedence when set. ttl=NNN header value is parsed but still deferred to v1.1 (returns 400 cache_override_not_implemented with a cache_rule_mode_not_implemented WARN on rules).

​

Provider matrix

​

Modes

​

respect (default)

​

disable

• Anthropic: recursive JSON-walk drops cache_control from messages[].content[], system[], and tools[].
• Gemini: drops cachedContent from the top-level body.
• OpenAI: no-op on wire (the provider’s automatic cache can’t be disabled client-side).

• Benchmarking cold-cost in A/B before deciding if a caching tier is worth turning on.
• Reproducing a cache-dependent bug on a cold call.
• A specific request must not hit cache for correctness.

​

force

• Anthropic (direct + Bedrock Claude + Vertex Claude): v1 shipped. Gateway injects cache_control: {type: "ephemeral"} on system[-1] + messages[-1].content[-1] when the caller hasn’t already set it. Implemented via a recursive JSON walk that targets the last element of each block array; no-double-inject invariant means client-set cache_control is preserved byte-for-byte.
• OpenAI, Azure: v1 shipped (as a no-op on wire). Their cache is automatic, so the request body is byte-identical to passthrough, but the rule still fires + langwatch.cache.rule_id + mode_applied=FORCE span attrs emit + gateway_cache_rule_hits_total{mode_applied="FORCE"} counter bumps. Operators get per-rule observability even on providers where force is semantically a no-op.
• Gemini, Vertex Gemini: v1 shipped. The gateway’s /v1beta/* passthrough route (see the Gemini CLI integration) accepts POST /v1beta/cachedContents for context-cache creation and forwards verbatim to generativelanguage.googleapis.com. Subsequent /v1beta/models/{model}:generateContent calls reference the cache by name via the cachedContent field. The same cached_content field is also lifted from /v1/chat/completions request bodies onto Bifrost.ChatParameters.ExtraParams so cross-shape OpenAI-SDK callers can reference an existing cache. Implicit prefix caching (auto-cache for >1024-token shared prefixes) requires a paid-tier Google account.

​

ttl=NNN

• Anthropic: chooses ephemeral (5 min) or persistent (1 h) based on the TTL range; anything above 3600 clamps to persistent.
• Gemini: sets the ttl field on cachedContent (depends on the v1.1 Gemini force work above).

​

Response headers

• X-LangWatch-Cache-Mode, the mode that was applied (respect, disable, force). Matches the internal Kind enum the Go gateway uses (services/gateway/internal/cacheoverride). Useful for tracing and A/B analysis: a client that sets X-LangWatch-Cache: disable should see X-LangWatch-Cache-Mode: disable on the response, independent of whether the provider reported a cache outcome.
• X-LangWatch-Cache, the cache outcome:
  • hit, provider served part or all of the prefix from cache.
  • miss, no cache hit.
  • bypass, mode was disable.
  • force, mode was force (v1.1, deferred with 400 cache_override_not_implemented in v1).

• cache_read_input_tokens, served from cache (Anthropic: ~10% of regular price).
• cache_creation_input_tokens, writing to cache (Anthropic: 125% of regular).
• input_tokens, regular cold tokens.

​

Cache rules

​

Shape

{
  "id": "rule_01HZX...",
  "priority": 100,
  "match": {
    "vk_tags": ["env=prod"],
    "model": "gpt-5-mini",
    "principal_id": null,
    "request_metadata": { "X-Request-Source": "internal-api" }
  },
  "action": {
    "mode": "force",
    "ttl": 300,
    "salt": "v2"
  },
  "enabled": true
}

​

Matchers

• vk_id, exact match on virtual-key id.
• vk_tags, every tag in the list must be on the VK (e.g. ["env=prod", "team=ml"]).
• vk_prefix, match on VK display prefix (vk-lw-01HZX9).
• principal_id, the user/service account making the call.
• model, exact match on the resolved model (post-alias).
• request_metadata, match on arbitrary client-set metadata headers (X-Request-Source, X-Customer-Tier, etc.).
• time_window, optional time-of-day, day-of-week constraint.

​

Actions

• mode: respect | disable | force, as above.
• ttl: NNN, seconds (only meaningful with force).
• salt: <string>, cache-key salt for A/B or version invalidation (for the gateway’s semantic cache, v1.1).

​

Evaluation

• First-match wins, sorted by priority descending.
• Rules evaluated once per request, AT auth-resolve time (bundled with VK config), so zero additional upstream latency.
• Rule hits exposed in observability via langwatch.cache.rule_id span attribute + gateway_cache_rule_hits_total{rule_id, mode, provider} counter.

​

Concrete example: “force 5-min cache on all prod internal-api calls”

{
  "priority": 100,
  "match": {
    "vk_tags": ["env=prod"],
    "request_metadata": { "X-Request-Source": "internal-api" }
  },
  "action": { "mode": "force", "ttl": 300 }
}

​

Enforcement ordering

1. Auth resolution (VK → bundle)
2. Rate limit (per-VK)
3. Cache override mode resolution (header > rule > VK default)
4. Cache override apply (strip, inject)
5. Blocked patterns (tools, MCP, URLs, models)
6. Body size cap
7. Guardrails (pre)
8. Upstream dispatch
9. Guardrails (post)
10. Budget debit

​

Why this is load-bearing

• Without caching: 40k × 100 = 4M input tokens/day at full price. Claude Opus: ~$60/day.
• With Anthropic ephemeral cache (5-min TTL, 1 write + ~99 reads): 1 write at $75/M + 99 × 40k reads at$6/M ≈ $27/day. 56% cheaper.
• With a gateway that reformats the payload and loses the cache: back to $60/day.

​

See also

• Caching Passthrough: the lower-level byte-for-byte invariant that underpins respect mode.
• Virtual Keys: per-VK default cache config.
• API Reference → Errors: cache_override_invalid, cache_override_not_implemented error shapes.
• Observability: the trace attributes for cache attribution.