Skip to main content
You’re already calling OpenAI, Anthropic, Bedrock, etc. from your application. You want to add governance, budgets, fallback, policy-rule patterns, per-engineer attribution, without rewriting everything. This cookbook shows the minimum-effort migration.

​
Before

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

​
After

const openai = new OpenAI({
  baseURL: "https://gateway.langwatch.ai/v1",
  apiKey: process.env.LANGWATCH_VK,
});
const anthropic = new Anthropic({
  baseURL: "https://gateway.langwatch.ai",
  apiKey: process.env.LANGWATCH_VK,
});
Two env-var changes. That’s the whole code migration if you’re happy with default gateway behaviour (no budget, cache-respect, fallback disabled). Read on for the policy additions.

​
Pre-migration checklist

  • LangWatch project provisioned (you already have one for tracing, reuse it).
  • API token with modelProviders:update + virtualKeys:create at the same ORGANIZATION scope. Use the LangWatch UI → Settings → API Tokens.
  • Access to the upstream provider accounts (you’ll rebind their existing keys inside LangWatch).

​
Step 1: Configure the providers you already use

For each provider (OpenAI, Anthropic, Bedrock, Azure, Vertex, Gemini) configure (or reuse) a ModelProvider at the organization scope. In the UI: Settings → Model Providers → Add Model Provider. Pick Scope = Organization so every project in the org inherits the credential. Then open each row and switch to the Advanced (Gateway) tab to set the gateway-only fields:
  • Rate limit (RPM / TPM / RPD) — caps the gateway applies before dispatch.
  • Fallback priority — used when the org’s default RoutingPolicy resolves more than one eligible MP (lower wins; tiebreak by createdAt).
  • Provider config (JSON) — region/deployment overrides (e.g. {"region": "us-east-1"}).
No separate “gateway provider” entity to mint. The credential itself is the only thing.
If you’ve never configured ModelProviders in LangWatch before, this is the same Settings page you’d use for prompt playground + evaluators — the gateway just reads from the same store.

​
Step 2: Mint your first virtual key

export LANGWATCH_API_KEY="lwp_..."

langwatch virtual-keys create \
  --name "dev-migration" \
  --env live \
  --scope ORG:acme \
  --format json | tee /tmp/vk.json

export LANGWATCH_VK=$(jq -r '.secret' /tmp/vk.json)
Save the secret. It’s shown exactly once. The VK lives at ORG:acme and inherits every ModelProvider visible from that scope. To pin a specific provider order, create a RoutingPolicy and pass --routing-policy <id>; otherwise the org’s default ordering applies.

​
Step 3: Flip the env vars in your app

Dev/staging first: 
# Before
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...

# After
OPENAI_API_KEY=$LANGWATCH_VK
OPENAI_BASE_URL=https://gateway.langwatch.ai/v1
ANTHROPIC_API_KEY=$LANGWATCH_VK
ANTHROPIC_BASE_URL=https://gateway.langwatch.ai
Redeploy. Every request now flows through the gateway.

​
Verify the migration is live

curl -sD- -o/dev/null \
  -H "Authorization: Bearer $LANGWATCH_VK" \
  -H "Content-Type: application/json" \
  -X POST https://gateway.langwatch.ai/v1/chat/completions \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"ping"}],"max_tokens":4}' | \
  grep -i x-langwatch-
You should see X-LangWatch-Request-Id, X-LangWatch-Trace-Id, X-LangWatch-Span-Id. Paste the request id into the LangWatch search bar, the full trace is already there.

​
Step 4: Add your first policy

The whole point of the migration. Pick the policy that maps to a real pain you have today:

​
Hard cap on engineering spend

langwatch gateway-budgets create \
  --name "engineering-monthly-cap" \
  --scope team --team team_eng... \
  --window month --limit 5000 --on-breach block
Every VK attached to engineering principals now shares the $5K/month envelope.

​
Automatic failover when OpenAI is flaky

Create a RoutingPolicy listing the ModelProvider ids in dispatch order (OpenAI primary, Anthropic backup), then point the VK at it: 
# Create a policy at org scope
langwatch routing-policies create \
  --scope ORG:acme \
  --name "openai-then-anthropic" \
  --strategy ordered \
  --model-provider mp_openai_org \
  --model-provider mp_anthropic_org \
  --format json | jq -r '.id'
# → rp_01HZX...

# Re-bind the VK to it
langwatch virtual-keys update vk_01HZX... \
  --routing-policy rp_01HZX...
When OpenAI throws a 503, the gateway re-dispatches the same prompt to Anthropic transparently. Your application sees a normal 200 response. Tune the fallback conditions (5xx / timeout / rate_limit / network_error) via the VK --config-json field; the model_aliases map still lives there for client-side name redirects.

​
Block destructive tools on engineer VKs

{
  "policy_rules": {
    "tools": { "deny": ["^shell\\.exec$", "^filesystem\\.write$"], "allow": null }
  }
}
Agent calls like shell.exec("rm -rf ~") get 403 tool_not_allowed before they reach the model, the model never generates the destructive call path in the first place.

​
Step 5: Wire trace propagation

If you were already using LangWatch SDKs for tracing, propagate the trace id into the gateway so requests nest under your existing trace (no double-cost-attribution): 
import langwatch

client = OpenAI(
    base_url="https://gateway.langwatch.ai/v1",
    api_key=os.environ["LANGWATCH_VK"],
    default_headers=langwatch.get_gateway_headers(),
)
See Python SDK → trace propagation and TypeScript SDK.

​
Step 6: Rotate the original upstream keys

After a week of running through the gateway, the original upstream keys (sk-proj-..., sk-ant-...) should no longer be used by any application. Rotate them in the provider console, this is the only time you need to touch upstream again. The gateway still has access via its own encrypted copies of the old keys (fetched from the LangWatch control plane), and will continue to serve traffic uninterrupted.

​
Troubleshooting

SymptomCauseFix
401 invalid_api_key on every requestVK env var not loaded; or VK revokedecho $LANGWATCH_VK to verify; check LangWatch UI for VK status
403 model_not_allowed on a model you used beforeVK’s models_allowed is set restrictivelyEdit VK to add the model, or remove the allowlist
Calls succeed but no trace in LangWatch UIScope-id mismatch between VK and where you’re lookinglangwatch virtual-keys get <id> → check scopes (and resolved project_id per vk-config-bundle rules — TEAM/ORG-scoped VKs file traces in the org’s internal_governance project)
Anthropic-side calls return 400 must provide max_tokensYou removed max_tokens because OpenAI doesn’t require itAnthropic requires it; keep it set
Response latency jumped by ~50 msFallback is running every call (primary is down)Check X-LangWatch-Fallback-Count, if > 0, fix the primary

​
What NOT to change

  • Request bodies: the gateway’s whole point is byte-for-byte passthrough. Don’t rewrite your payloads.
  • SDKs: the official OpenAI, Anthropic SDKs work unchanged. You don’t need the LangWatch SDK for gateway integration (only for trace propagation).
  • Streaming handlers: SSE passthrough is byte-identical post-first-chunk. Your streaming code should keep working.

​
Rollback plan

If the gateway misbehaves, flip the env vars back: 
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_API_KEY=sk-proj-...    # original key
Redeploy. You’re back to direct provider calls. Traces in LangWatch stop populating (the trace propagation + gateway spans go away), but your app works. Plan the first rollout behind a feature flag for this exact reason, 10% traffic through the gateway for a day before going 100%.

​
See also