Migrating from direct provider calls

You’re already calling OpenAI, Anthropic, Bedrock, etc. from your application. You want to add governance, budgets, fallback, policy-rule patterns, per-engineer attribution, without rewriting everything. This cookbook shows the minimum-effort migration.

Before

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

After

const openai = new OpenAI({
  baseURL: "https://gateway.langwatch.ai/v1",
  apiKey: process.env.LANGWATCH_VK,
});
const anthropic = new Anthropic({
  baseURL: "https://gateway.langwatch.ai",
  apiKey: process.env.LANGWATCH_VK,
});

Two env-var changes. That’s the whole code migration if you’re happy with default gateway behaviour (no budget, cache-respect, fallback disabled). Read on for the policy additions.

Pre-migration checklist

LangWatch project provisioned (you already have one for tracing, reuse it).
API token with modelProviders:update + virtualKeys:create at the same ORGANIZATION scope. Use the LangWatch UI → Settings → API Tokens.
Access to the upstream provider accounts (you’ll rebind their existing keys inside LangWatch).

Step 1: Configure the providers you already use

For each provider (OpenAI, Anthropic, Bedrock, Azure, Vertex, Gemini) configure (or reuse) a ModelProvider at the organization scope. In the UI: Settings → Model Providers → Add Model Provider. Pick Scope = Organization so every project in the org inherits the credential. Then open each row and switch to the Advanced (Gateway) tab to set the gateway-only fields:

Rate limit (RPM / TPM / RPD) — caps the gateway applies before dispatch.
Fallback priority — used when the org’s default RoutingPolicy resolves more than one eligible MP (lower wins; tiebreak by createdAt).
Provider config (JSON) — region/deployment overrides (e.g. {"region": "us-east-1"}).

No separate “gateway provider” entity to mint. The credential itself is the only thing.

If you’ve never configured ModelProviders in LangWatch before, this is the same Settings page you’d use for prompt playground + evaluators — the gateway just reads from the same store.

Step 2: Mint your first virtual key

export LANGWATCH_API_KEY="lwp_..."

langwatch virtual-keys create \
  --name "dev-migration" \
  --env live \
  --scope ORG:acme \
  --format json | tee /tmp/vk.json

export LANGWATCH_VK=$(jq -r '.secret' /tmp/vk.json)

Save the secret. It’s shown exactly once. The VK lives at ORG:acme and inherits every ModelProvider visible from that scope. To pin a specific provider order, create a RoutingPolicy and pass --routing-policy <id>; otherwise the org’s default ordering applies.

Step 3: Flip the env vars in your app

Dev/staging first:

# Before
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...

# After
OPENAI_API_KEY=$LANGWATCH_VK
OPENAI_BASE_URL=https://gateway.langwatch.ai/v1
ANTHROPIC_API_KEY=$LANGWATCH_VK
ANTHROPIC_BASE_URL=https://gateway.langwatch.ai

Redeploy. Every request now flows through the gateway.

Verify the migration is live

curl -sD- -o/dev/null \
  -H "Authorization: Bearer $LANGWATCH_VK" \
  -H "Content-Type: application/json" \
  -X POST https://gateway.langwatch.ai/v1/chat/completions \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"ping"}],"max_tokens":4}' | \
  grep -i x-langwatch-

You should see X-LangWatch-Request-Id, X-LangWatch-Trace-Id, X-LangWatch-Span-Id. Paste the request id into the LangWatch search bar, the full trace is already there.

Step 4: Add your first policy

The whole point of the migration. Pick the policy that maps to a real pain you have today:

Hard cap on engineering spend

langwatch gateway-budgets create \
  --name "engineering-monthly-cap" \
  --scope team --team team_eng... \
  --window month --limit 5000 --on-breach block

Every VK attached to engineering principals now shares the $5K/month envelope.

Automatic failover when OpenAI is flaky

Create a RoutingPolicy listing the ModelProvider ids in dispatch order (OpenAI primary, Anthropic backup), then point the VK at it:

# Create a policy at org scope
langwatch routing-policies create \
  --scope ORG:acme \
  --name "openai-then-anthropic" \
  --strategy ordered \
  --model-provider mp_openai_org \
  --model-provider mp_anthropic_org \
  --format json | jq -r '.id'
# → rp_01HZX...

# Re-bind the VK to it
langwatch virtual-keys update vk_01HZX... \
  --routing-policy rp_01HZX...

When OpenAI throws a 503, the gateway re-dispatches the same prompt to Anthropic transparently. Your application sees a normal 200 response. Tune the fallback conditions (5xx / timeout / rate_limit / network_error) via the VK --config-json field; the model_aliases map still lives there for client-side name redirects.

Block destructive tools on engineer VKs

{
  "policy_rules": {
    "tools": { "deny": ["^shell\\.exec$", "^filesystem\\.write$"], "allow": null }
  }
}

Agent calls like shell.exec("rm -rf ~") get 403 tool_not_allowed before they reach the model, the model never generates the destructive call path in the first place.

Step 5: Wire trace propagation

If you were already using LangWatch SDKs for tracing, propagate the trace id into the gateway so requests nest under your existing trace (no double-cost-attribution):

import langwatch

client = OpenAI(
    base_url="https://gateway.langwatch.ai/v1",
    api_key=os.environ["LANGWATCH_VK"],
    default_headers=langwatch.get_gateway_headers(),
)

See Python SDK → trace propagation and TypeScript SDK.

Step 6: Rotate the original upstream keys

After a week of running through the gateway, the original upstream keys (sk-proj-..., sk-ant-...) should no longer be used by any application. Rotate them in the provider console, this is the only time you need to touch upstream again. The gateway still has access via its own encrypted copies of the old keys (fetched from the LangWatch control plane), and will continue to serve traffic uninterrupted.

Troubleshooting

Symptom	Cause	Fix
`401 invalid_api_key` on every request	VK env var not loaded; or VK revoked	`echo $LANGWATCH_VK` to verify; check LangWatch UI for VK status
`403 model_not_allowed` on a model you used before	VK’s `models_allowed` is set restrictively	Edit VK to add the model, or remove the allowlist
Calls succeed but no trace in LangWatch UI	Scope-id mismatch between VK and where you’re looking	`langwatch virtual-keys get <id>` → check `scopes` (and resolved `project_id` per vk-config-bundle rules — TEAM/ORG-scoped VKs file traces in the org’s `internal_governance` project)
Anthropic-side calls return `400 must provide max_tokens`	You removed `max_tokens` because OpenAI doesn’t require it	Anthropic requires it; keep it set
Response latency jumped by ~50 ms	Fallback is running every call (primary is down)	Check `X-LangWatch-Fallback-Count`, if > 0, fix the primary

What NOT to change

Request bodies: the gateway’s whole point is byte-for-byte passthrough. Don’t rewrite your payloads.
SDKs: the official OpenAI, Anthropic SDKs work unchanged. You don’t need the LangWatch SDK for gateway integration (only for trace propagation).
Streaming handlers: SSE passthrough is byte-identical post-first-chunk. Your streaming code should keep working.

Rollback plan

If the gateway misbehaves, flip the env vars back:

OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_API_KEY=sk-proj-...    # original key

Redeploy. You’re back to direct provider calls. Traces in LangWatch stop populating (the trace propagation + gateway spans go away), but your app works. Plan the first rollout behind a feature flag for this exact reason, 10% traffic through the gateway for a day before going 100%.

Get Started

SDK Integration

Coding CLI Integrations

Virtual Keys & Budgets

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

Migrating from direct provider calls

Before

After

Pre-migration checklist

Step 1: Configure the providers you already use

Step 2: Mint your first virtual key

Step 3: Flip the env vars in your app

Verify the migration is live

Step 4: Add your first policy

Hard cap on engineering spend

Automatic failover when OpenAI is flaky

Block destructive tools on engineer VKs

Step 5: Wire trace propagation

Step 6: Rotate the original upstream keys

Troubleshooting

What NOT to change

Rollback plan

See also

​Before

​After

​Pre-migration checklist

​Step 1: Configure the providers you already use

​Step 2: Mint your first virtual key

​Step 3: Flip the env vars in your app

​Verify the migration is live

​Step 4: Add your first policy

​Hard cap on engineering spend

​Automatic failover when OpenAI is flaky

​Block destructive tools on engineer VKs

​Step 5: Wire trace propagation

​Step 6: Rotate the original upstream keys

​Troubleshooting

​What NOT to change

​Rollback plan

​See also

Before

After

Pre-migration checklist

Step 1: Configure the providers you already use

Step 2: Mint your first virtual key

Step 3: Flip the env vars in your app

Verify the migration is live

Step 4: Add your first policy

Hard cap on engineering spend

Automatic failover when OpenAI is flaky

Block destructive tools on engineer VKs

Step 5: Wire trace propagation

Step 6: Rotate the original upstream keys

Troubleshooting

What NOT to change

Rollback plan

See also