The 8 Best LLM Gateways in 2026: Compared for Production

In 2024, an LLM gateway was a convenience: one API shape for many providers, so you could swap models without rewriting code. In 2026, it is the control point of the whole AI stack. The clearest signal came this spring, when Palo Alto Networks, a security company, acquired Portkey, a gateway company. Gateways stopped being plumbing and became the place where AI spend, security, and observability converge.

Two things changed. First, model traffic stopped coming only from your applications: coding agents like Claude Code and Cursor now generate a large share of enterprise token spend, and they need governed credentials just like production services do. Second, the questions leadership asks moved from "which model is best?" to "who is spending this, on what, and can we stop it?", the questions we hear in every enterprise working session (we collected them in our Agent Governance Playbook).

This guide compares the eight LLM gateways that matter in 2026, in enough depth to actually choose one. We build one of them, so we will tell you exactly where ours is strongest and where another tool is the better fit.

What is an LLM gateway?

An LLM gateway is a proxy that sits between your applications and model providers. Every request flows through one endpoint, which gives the gateway a unique position to do things no SDK or dashboard can:

• One API, many models. OpenAI-compatible (and increasingly Anthropic-compatible) endpoints mean switching from GPT-5 to Claude to Gemini is a config change, not a rewrite.
• Credential control. Applications hold gateway-issued keys, not raw provider keys, so a leaked key is revocable in seconds and scoped to a budget.
• Spend enforcement. Budgets and rate limits applied in the request path, before the money is spent, not discovered on the invoice.
• Reliability. Retries, fallback chains across providers, and caching.
• Security and quality. Guardrails that inspect requests and responses inline: PII, prompt injection, off-policy tool calls.
• Attribution. Every request tagged with who made it, from which team, at what cost.

The gateway pattern has one honest cost: it puts a hop in front of every model call. That makes two numbers non-negotiable when you evaluate one: the gateway's own processing overhead, and what happens to your traffic when the gateway's control plane has a bad day.

How we compared these LLM gateways

We weighted five dimensions, in the order enterprise buyers actually ask about them in 2026:

1. Governance depth: per-key budgets, credential lifecycle, guardrails, audit trails, and whether coding agents can be governed alongside apps.
2. Observability: does the gateway produce traces you can debug and evaluate from, or counters you can only glance at?
3. Performance: gateway-side overhead under sustained load, and streaming fidelity (tool-call deltas survive, or break).
4. Deployment freedom: managed, self-hosted, or both; and what still works when the control plane is unreachable.
5. Pricing shape: subscription, token markup, or infrastructure cost, and how it scales with volume.

Gateway	Standout strength	Open source	Deployment	Pricing shape
LangWatch	Governance plus evals and observability in one platform	Apache-2.0	Cloud, self-hosted (Helm), hybrid	Platform plan, no token markup
LiteLLM	The default open-source proxy, 100+ providers	Core is OSS	Self-hosted, enterprise cloud	Free OSS, paid enterprise
Portkey	Enterprise gateway, now part of Palo Alto Prisma AIRS	No	Managed, enterprise options	Subscription, enterprise
OpenRouter	300+ models behind one key, zero setup	No	Managed only	Pay per token plus credit fee
Kong AI Gateway	AI plugins on proven API-gateway infrastructure	Kong Gateway is OSS	Self-hosted, Konnect cloud	OSS free, Konnect tiers
Vercel AI Gateway	Frictionless for AI SDK apps, zero markup	No	Managed only	Pay per token, BYOK free
Cloudflare AI Gateway	Edge caching and analytics, generous free tier	No	Managed edge	Free core, paid logging tiers
Amazon Bedrock	AWS-native model access with IAM governance	No	AWS managed	Per-token, AWS billing

The 8 best LLM gateways in 2026

1. LangWatch

LangWatch's AI Gateway is built for the moment the gateway stops being an application concern and becomes an organizational one. It is a single OpenAI- and Anthropic-compatible endpoint that every application, SDK, and coding CLI in the organization sends traffic to, and it treats that position in front of every model call as a responsibility: policy, budgets, guardrails, attribution, and a full trace applied to each request, with routing behavior that is explicit rather than magical.

Fast enough to sit in front of everything: the gateway is written in Go and measures roughly 11 microseconds of gateway-side overhead at 5,000 sustained requests per second. That number is the difference between a gateway you route everything through and a gateway you quietly bypass for latency-sensitive paths. Streaming preserves the tool-call deltas that agentic workloads depend on, which is why Claude Code and Codex work through it unmodified.

Routing you can reason about: fallback across providers (OpenAI to Anthropic to Bedrock on 5xx, timeouts, and rate limits) is an explicit per-key choice rather than a silent default, and client errors never trigger it, so a bad request fails loudly instead of burning three providers. Anthropic cache_control markers pass through byte for byte, so routing never quietly costs you the 90% prompt-cache discount, and switching every internal app to a different provider is a model_aliases change on the keys involved, zero code changes.

Virtual keys and hierarchical budgets: instead of sharing provider keys, you mint virtual keys scoped to a project, team, or person: show-once secrets, rotatable, revocable within 60 seconds. Budgets enforce spend at any scope (organization, team, project, key, or individual end user) with soft-warn or hard-block semantics and periods from a minute to a full grant. "Give the support team a key they cannot leak, capped at $500 a month" is a two-minute task.

Guardrails that are actually evaluators: because the gateway is part of the LangWatch platform, inline guardrails run the same evaluators you use for offline testing: PII detection, prompt-injection screening, policy checks, applied to the request, the response, or each streaming chunk, with allow, block, or redact decisions. Tool calls, MCP server references, and outbound URLs can be allowlisted or blocked by pattern before a request ever leaves your infrastructure.

Coding agents are first-class citizens: engineers run one login command and get a personal virtual key; Claude Code, Codex, Cursor, Gemini CLI, opencode, and Aider route through the gateway with per-person attribution and budgets. For the growing share of enterprises whose biggest AI line item is coding assistants, this is the difference between governing your AI spend and governing some of it.

Where it fits: teams that want the gateway, the traces, and the evaluations in one place rather than three vendors. Every request emits a full OpenTelemetry trace with cost, cache breakdown, and attribution into the same platform that runs your evals and agent simulations. It ships self-hosted via Helm (and keeps serving traffic if the control plane goes down), hybrid, or in EU, US, UK, and APAC cloud, with ISO 27001, GDPR, RBAC, SSO/SCIM, and audit logging. If you only want a thin router and nothing else, a lighter tool below may be all you need.

2. LiteLLM

LiteLLM is the open-source workhorse of this category: a Python proxy and SDK that normalizes 100+ providers behind an OpenAI-compatible interface. If a model API exists, LiteLLM probably speaks it, and that coverage is why it shows up inside so many other products (and reference architectures) as the translation layer.

Run it yourself, own everything: the proxy self-hosts anywhere Python runs, with virtual keys, team budgets, rate limits, and spend tracking built in. Config lives in YAML, callbacks stream usage data to the logging or observability stack of your choice, and the community moves fast when providers ship new API surface.

The honest tradeoffs: a Python hot path means overhead measured in milliseconds rather than microseconds, which matters at high sustained RPS. Governance features exist but are operator-grade: you assemble budgets, alerting, and dashboards yourself, and observability is delegated to whatever you wire the callbacks into. Enterprise features (SSO, audit logs) sit behind a paid tier.

Where it fits: platform teams with the appetite to operate their own gateway and integrate it into an existing observability stack. It is also a strong escape hatch: because it is OpenAI-compatible on both sides, migrating off it later is cheap.

3. Portkey (now Palo Alto Networks)

Portkey built one of the most complete managed gateways of the last few years: unified API, configurable routing and fallbacks, semantic caching, request logging, prompt management, and guardrails. In May 2026, Palo Alto Networks completed its acquisition of Portkey, folding the gateway into its Prisma AIRS platform as the control plane for securing AI agents.

What it does well: mature routing primitives (weighted load balancing, conditional routing, retries with backoff), a guardrails catalog, and enterprise controls that predate most competitors. Teams already running Portkey report solid reliability at scale, and the Palo Alto deal validates the category's security importance.

The honest tradeoffs: acquisitions reshape roadmaps. Portkey's center of gravity is now a security suite, and buyers should expect packaging, pricing, and integration priorities to orient around Prisma AIRS and Palo Alto's install base. If you are a Palo Alto shop, that is a feature. If you are not, evaluate what standalone investment looks like over the next two years.

Where it fits: enterprises standardizing on Palo Alto for AI security who want gateway controls from the same vendor.

4. OpenRouter

OpenRouter is the fastest way to get access to the whole model landscape: 300+ models from dozens of labs behind one OpenAI-compatible API and one balance. For evaluation sprints, side projects, and products that let users pick their model, nothing else is this frictionless.

What it does well: instant availability of new models (often day one), automatic failover between providers hosting the same open-weight model, transparent per-token pass-through pricing with a small fee on credits, and BYOK support. Its public leaderboards have become a de facto market-share signal for the industry.

The honest tradeoffs: OpenRouter is a marketplace, not a governance layer. Organization-level budgets, per-developer attribution, guardrails, audit trails, and self-hosting are not the product. Traffic terminates in its US cloud, which is a conversation with your compliance team if you handle regulated data.

Where it fits: developers and product teams that value model breadth and zero setup over enterprise controls, or as the experimentation tier behind a governed gateway.

5. Kong AI Gateway

Kong took the API gateway enterprises already trust and taught it to speak LLM. AI plugins add a unified provider interface, token-aware rate limiting, semantic caching, semantic routing, and prompt guarding on top of Kong Gateway's battle-tested proxy, with management through Konnect.

What it does well: if your platform team already operates Kong, the AI Gateway inherits your existing deployment model, plugin ecosystem, RBAC, and operational muscle memory. Multi-LLM routing and caching policies live next to the policies protecting the rest of your APIs, and recent releases extended coverage toward MCP and agent-to-agent traffic.

The honest tradeoffs: Kong is infrastructure, not an LLM product. There are no evaluations, no trace-level debugging of model behavior, and cost attribution is at the API-analytics level rather than the "which prompt, which user, which conversation" level. Teams without existing Kong expertise face a real learning curve.

Where it fits: organizations with Kong already in production that want LLM traffic governed by the same infrastructure, with an LLM-native observability layer on top.

6. Vercel AI Gateway

Vercel's AI Gateway is the path of least resistance for teams building with the AI SDK: change one line and your app can reach hundreds of models with unified billing, automatic failover, and zero markup on tokens, including with your own provider keys.

What it does well: the developer experience is excellent, budgets and spend monitoring cover the essentials, and zero markup makes the pricing math simple. For Next.js and AI SDK teams it removes an entire category of setup work, and latency from Vercel-hosted apps is minimal.

The honest tradeoffs: it is an application-level convenience, not an organization-level control plane. Governance is scoped to Vercel projects and teams, guardrails and evaluations are not the product, and traffic from systems outside the Vercel ecosystem is an awkward fit. No self-hosting.

Where it fits: product teams shipping on Vercel that want model flexibility and clean billing without running infrastructure.

7. Cloudflare AI Gateway

Cloudflare's AI Gateway gives you caching, rate limiting, retries, fallback, logging, and cost analytics at the edge, in front of most major providers, with a generous free tier. For the price of a base-URL change you get a global cache and a dashboard.

What it does well: edge termination keeps added latency low worldwide, response caching genuinely cuts bills on repetitive traffic, and it composes naturally with Workers AI and the rest of the Cloudflare stack. The free tier makes it the cheapest way to get basic visibility on LLM traffic.

The honest tradeoffs: it is analytics, not observability: you see requests, costs, and cache hits, not evaluable traces of multi-step agent behavior. Governance primitives (credential lifecycle, per-team budgets, guardrails, audit) are thin, and everything lives in Cloudflare's cloud.

Where it fits: teams already on Cloudflare that want caching and cost visibility with near-zero effort, often in front of a deeper governance layer rather than instead of one.

8. Amazon Bedrock

Bedrock is AWS's answer to the gateway question: a managed service exposing a curated catalog of foundation models (Anthropic, Meta, Mistral, Amazon's own Nova line, and more) through AWS APIs, with IAM for access control, CloudWatch for monitoring, Guardrails for content policy, and cross-region inference for resilience.

What it does well: if your organization runs on AWS, Bedrock means model access inherits the governance you already have: IAM policies, PrivateLink, KMS encryption, CloudTrail audit. Procurement is a line on the AWS bill instead of a new vendor, which shortens enterprise adoption dramatically.

The honest tradeoffs: it is a walled garden by design. OpenAI's flagship API models are not in the catalog, multi-cloud strategies need a second layer anyway, and the developer experience (model IDs, regional availability, quota management) is AWS-shaped rather than OpenAI-compatible. Token-level cost attribution per team or user still requires building on top of tagging and CloudWatch.

Where it fits: AWS-committed enterprises whose model needs are covered by the catalog and whose governance already lives in IAM.

Which LLM gateway should you choose?

Match the tool to your operating model, not to feature checklists:

• You need organization-wide governance, budgets per team, guardrails inline, coding agents included: LangWatch, self-hosted or cloud.
• You want open source and full operational control, and you have the platform team to run it: LiteLLM, with an observability layer wired into its callbacks.
• You are standardizing AI security on Palo Alto: Portkey inside Prisma AIRS.
• You want every model with zero setup: OpenRouter, and add governance when the traffic starts to matter.
• Your platform team already runs Kong: Kong AI Gateway for transport, plus LLM-native observability on top.
• You ship on Vercel with the AI SDK: Vercel AI Gateway.
• You want caching and cost visibility today, for free: Cloudflare AI Gateway.
• You are AWS-first and the catalog covers you: Amazon Bedrock.

One more honest note, since we made the same point in our Agent Governance Playbook: a gateway can only govern the traffic that flows through it. Coding assistants on seat-based subscriptions and vendor copilots often cannot be routed through any proxy, which is why LangWatch pairs the gateway with pull-mode telemetry for the tools a gateway cannot see. Observe and alert everywhere, enforce where the gateway sits in the path.

Try a governed LLM gateway

The fastest way to evaluate LangWatch's gateway is to route one application or one coding CLI through it: swap base_url and key, and watch the traces, budgets, and attribution appear. The quickstart takes about five minutes, the platform is open source, and the gateway ships in the same Helm chart as everything else if you self-host.

If you are choosing a gateway for an organization rather than an app, book a working session: bring two teams' traffic, leave with both governed, costed, and traced in one pane.