Streaming

Which endpoints stream

Endpoint	How you ask for a stream
`POST /v1/chat/completions`	`"stream": true` in the body
`POST /v1/messages`	`"stream": true` in the body
`POST /v1/responses`	`"stream": true` in the body
`POST /v1beta/models/{model}:streamGenerateContent`	The method name. The Gemini SDKs call it for you
`POST /v1beta/models/{model}:streamGenerateAnswer`	The method name

/v1/embeddings, /v1/audio/*, /v1/images/* and /v1/models return one response and never stream. On the image routes a stream of true, or a partial_images above zero, answers 400 before any provider is contacted.

Start a stream

The base URL for an OpenAI-style SDK is https://gateway.langwatch.ai/v1. The Anthropic and Gemini SDKs append their own version segment, so give them https://gateway.langwatch.ai.

OpenAI
Anthropic

from openai import OpenAI

client = OpenAI(base_url="https://gateway.langwatch.ai/v1", api_key="vk-lw-...")

for chunk in client.chat.completions.create(
    model="openai/gpt-5-mini",
    messages=[{"role": "user", "content": "count to 5"}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="")

import anthropic

client = anthropic.Anthropic(base_url="https://gateway.langwatch.ai", api_key="vk-lw-...")

with client.messages.stream(
    model="anthropic/claude-haiku-4-5-20251001",
    max_tokens=64,
    messages=[{"role": "user", "content": "count to 5"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="")

What the frames look like

The frame shape follows the endpoint you called, not the provider that served the request. POST /v1/chat/completions sends OpenAI chunk objects, one per data: line, and closes with data: [DONE].

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"1"}}]}

data: [DONE]

POST /v1/messages sends the Anthropic Messages event union: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop. When an Anthropic credential serves the request, the provider’s own SSE bytes reach the client unchanged. When another provider serves it, the gateway builds the same event union from that provider’s response. POST /v1/responses sends the OpenAI Responses events: response.created, response.output_text.delta, response.completed. The /v1beta Gemini routes forward the provider’s bytes unchanged. Tool-call deltas arrive in the endpoint’s own shape and keep their order. On /v1/messages a tool call is a content_block_start with "type": "tool_use", then input_json_delta frames whose partial_json strings concatenate into the argument object, then content_block_stop. On /v1/chat/completions the same call arrives as delta.tool_calls[] entries whose function.arguments strings concatenate.

When a stream fails after it opened

The gateway writes 200 OK and the text/event-stream headers before the first frame, so a failure after that point cannot change the status. The gateway ends the stream with an event: error frame instead.

event: error
data: {"type":"error","error":{"type":"provider_error","message":"upstream connection closed"}}

error.type carries the provider’s own discriminant when the provider named one, for example insufficient_quota or overloaded_error. For a failure the gateway raised, it carries the gateway error code. When the provider sent an error event of its own, that event body is forwarded unchanged instead of the frame above. Read error.type to decide what to do next. A provider failure is worth a retry after a backoff. The error index for every gateway code is on Troubleshooting.

When a guardrail stops a stream

A stream chunk guardrail runs on every chunk before the gateway forwards it. On a fail verdict the gateway closes the upstream stream at that chunk and forwards no further frame. The caller keeps the chunks already sent and gets no event: error frame, because the guardrail stops the stream rather than raising an error on it. Stream chunk guardrails always fail open. The gateway waits 50 milliseconds for the verdict; a timeout or an evaluator error lets the chunk through and records langwatch.guardrail.stream_chunk_fail_open with the cause on the span. A fail verdict blocks. A modify verdict does not rewrite the chunk. Also check: Guardrails for the directions, the failure modes and the setup steps.

Fallback on a stream

The gateway walks the key’s fallback chain while it opens the stream. Once the provider accepts the request and the first frame is written, no other credential can take over: splicing two providers into one stream would give the caller mismatched tool-call ids and double-counted tokens. A retryable failure before the first frame is therefore invisible to the caller. One stream arrives, from whichever credential accepted the request. X-LangWatch-Fallback-Count reports how many credentials were tried and skipped; the header is absent when the first one worked. A failure after the first frame ends the stream with the error frame above. Send the request again to walk the chain from the start. Also check: Fallback Chains.

Token usage on a stream

Providers report usage at the end of a stream, in their own shape.

Endpoint and provider	Where the usage arrives
`/v1/chat/completions` on OpenAI, Azure OpenAI, an OpenAI-compatible custom endpoint	The final data-only chunk, and only when the body carries `stream_options: {"include_usage": true}`
`/v1/messages`, any provider	`message_start` carries the input and cache counters, `message_delta` carries the running `output_tokens`
`/v1beta` on Gemini or Vertex	`usageMetadata` on the chunks that carry it

The gateway keeps the last non-zero counter it sees for each bucket, so an empty usage block late in the stream does not overwrite a real number. On /v1/chat/completions to OpenAI, Azure OpenAI or a custom OpenAI-compatible endpoint, the gateway sets stream_options.include_usage to true when the body does not carry it. Without the flag those providers report no usage at all, and the request debits zero tokens. A body that sets the flag itself, to true or to false, is left as sent. You can still set it yourself to keep the call explicit:

for chunk in client.chat.completions.create(
    model="openai/gpt-5-mini",
    messages=[{"role": "user", "content": "count to 5"}],
    stream=True,
    stream_options={"include_usage": True},
):
    pass

On /v1/chat/completions, a stream that closes with zero total tokens carries one more frame before data: [DONE]:

event: warning
data: {"warning":"provider_did_not_report_usage_on_stream"}

That stream debits zero tokens, so it passes budget enforcement without being counted. The gateway_streaming_usage_missing_total metric counts these per provider and model. Alert on any positive rate. See Observability.

Requests without `stream`

A request without stream: true returns one JSON body. The gateway runs the response guardrails on the full body, and the fallback chain stays open for the whole call because the caller sees only the response that succeeded.

A proxy in front of a self-hosted gateway must not buffer responses. With buffering on, every frame is held until the provider finishes and the caller gets the whole stream at once. On an nginx ingress set nginx.ingress.kubernetes.io/proxy-buffering: "off", and raise the read timeout above the longest stream you allow. See DNS and TLS.

Also check: Parameter mapping for what happens to a parameter the target provider has no field for, Observability for the span attributes and the metrics.

Get Started

SDK Integration

Virtual Keys & Budgets

Billing & Metering

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

Which endpoints stream

Start a stream

What the frames look like

When a stream fails after it opened

When a guardrail stops a stream

Fallback on a stream

Token usage on a stream

Requests without `stream`

​Which endpoints stream

​Start a stream

​What the frames look like

​When a stream fails after it opened

​When a guardrail stops a stream

​Fallback on a stream

​Token usage on a stream

​Requests without stream

Which endpoints stream

Start a stream

What the frames look like

When a stream fails after it opened

When a guardrail stops a stream

Fallback on a stream

Token usage on a stream

Requests without `stream`