Skip to main content

​
LangWatch Instant Preview or your own model

Instant Evals run on LangWatch Instant Preview by default, billed as in Limits and cost. You can pick a decision model in your own provider account instead, per organization, team or project. Your provider then bills you for each row, and LangWatch does not.
This is off by default. Ask LangWatch to turn it on for your organization. Until then, the Instant Evals column offers LangWatch Instant Preview only.

​
The models you can pick

OpenAI Decisions API and Azure OpenAI Decisions API need gpt-6-luna on that account. The other models need no OpenAI account.

​
Set it up

1

Add the provider

In Settings > Model Providers, click Add Model Provider, pick the provider from the table and enter its key. Scope it to each organization, team or project that should use it. See Model Providers.
2

Pick the model

In Default Models on the same page, add or edit a config and pick its scopes. In the Instant Evals picker, choose your model and save.
Edit default models drawer for the ACME organization, with the Instant Evals picker open on LangWatch Instant Preview and OpenAI Decisions API
3

Run

Start a run as usual. The new model judges the next rows within five seconds.
The config’s Instant Evals column then shows your model.
Default models table with the ACME config, whose Instant Evals column shows OpenAI Decisions API
The picker offers a model only when every scope of the config holds an enabled key for it. An organization config sees only keys saved on the organization, not a key one project holds. A config on two projects sees only what both hold. As for other default models, a project config beats a team config, which beats an organization config. A provider key saved on the project beats one on its team or organization.

​
Azure

Both Azure models use the Azure OpenAI provider, with its API key (AZURE_OPENAI_API_KEY) and endpoint (AZURE_OPENAI_ENDPOINT).
Azure OpenAI provider drawer with Use API Gateway off, the endpoint https://my-resource.openai.azure.com and the decisions deployment gpt-6-luna-eu
  • Endpoint (AZURE_OPENAI_ENDPOINT): your resource URL, such as https://my-resource.openai.azure.com. LangWatch takes the resource name from it and calls https://my-resource.openai.azure.com/openai/v1/decisions or, for Microsoft Decision 1, https://my-resource.services.ai.azure.com/providers/microsoft/v1/systemone. Hosts on openai.azure.com, cognitiveservices.azure.com and services.ai.azure.com work. Other hosts, http, custom ports and API Management gateways are refused, so the key stays inside Azure.
  • Decisions deployment (AZURE_OPENAI_DECISIONS_DEPLOYMENT): your gpt-6-luna deployment name. Leave empty if it is gpt-6-luna.
  • Microsoft Decision 1 deployment (AZURE_DECISION_1_DEPLOYMENT): your Microsoft Decision 1 deployment name on Azure AI Foundry. Leave empty if it keeps the model’s name.
Keep Use API Gateway off. With it on, the endpoint and deployment fields are hidden and every row is skipped as classifier_key_invalid. Deploy the model on your resource before you pick it, in the Azure portal, Azure AI Foundry or the Azure CLI:
Without the deployment, every row is skipped as classifier_not_configured.

​
When your model cannot answer

A run on your own model never falls back to LangWatch Instant Preview. Rows it cannot judge are marked skipped with the reason in error, and the run carries on. List them with langwatch instant-eval results <run-id> --status skipped. Rate limits and outages are retried before the row is skipped. Refused keys and missing models are not.

​
What changes on your own model

  • Cost. Your provider bills you. LangWatch does not, and the free budget does not apply.
  • Limits. The row, question and text limits still apply.
  • Data. Each row’s text and your questions go to your provider, under your agreement with it. LangWatch Instant Preview never sees them.
  • Search bar. Still runs on LangWatch Instant Preview.
Last modified on November 4, 2026