# Introducing Instant Evals: evaluate your entire production history

Run any eval you can think of, on all your production trace history. Cheap, fast, and at scale.

*By Rogerio Chaves · September 22, 2026*

Canonical: https://langwatch.ai/blog/introducing-instant-evals

![Introducing Instant Evals: evaluate your entire production history](https://langwatch.ai/blog/instant-evals-cover.webp)

Today we're launching [Instant Evals](https://langwatch.ai/instant-evals) in LangWatch: evals you run over your entire production history, any time you have a question about it. 10,000 conversations get judged in about a minute, for \$0.32.

```bash
npx langwatch instant-eval run "customer sounds frustrated" --last 30d
```

Until now, asking a new question over every trace meant paying an LLM judge to read each one, about \$31 and ten minutes per 10,000 conversations with Claude Opus 5. So everyone samples, and the sample misses the one conversation in five hundred where the agent promised a refund you don't give. Instant Evals reads all of them, at about a hundredth of the cost.

## Accuracy above frontier models, at a fraction of the time and cost

We took 300 synthetic support conversations, labelled each one by hand for whether the customer was frustrated, and asked Claude Opus 5, GPT 5.6, GPT 5 mini and Instant Evals the same question. Time and cost are scaled to 10,000 conversations.

![Two bar charts. Cost to judge 10,000 conversations: Claude Opus 5 \$31.13, GPT 5.6 \$25.28, GPT 5 mini \$3.93, Instant Evals \$0.32. Time to judge 10,000 conversations: Claude Opus 5 10 min, GPT 5.6 11 min, GPT 5 mini 25 min, Instant Evals 1 min 13 s. Agreement with the human labels: 91%, 88%, 90% and 97%](https://langwatch.ai/blog/instant-evals-benchmark.webp)

| Judge | Agreement with the human labels | Cost for 10,000 | Time for 10,000 |
|---|---|---|---|
| Claude Opus 5 | 91% | \$31.13 | 10 min |
| GPT 5.6 | 88% | \$25.28 | 11 min |
| GPT 5 mini | 90% | \$3.93 | 25 min |
| **Instant Evals** | **97%** | **\$0.32** | **1 min 13 s** |

That is one yes or no question on one dataset, and your question on your data can land differently, so check it before you trust the rate. `sample` prints a few judged rows with the text the judge read next to the verdict it gave:

```bash
langwatch instant-eval sample <id> -n 5
```

This is only possible thanks to Jev, the new model TypeSafe just launched. It's a small model built for one job, reading a text and answering a question about it, and it answers with a calibrated probability, so a 0.8 threshold means what it says. Small judge models like Jev are a new class that's here to stay, with alternatives already out there. Evals are our specialty at LangWatch, so we'll keep testing them and always run your evals on the most accurate and fastest one, which today is Jev. We run it for you, no provider keys needed.

## Your production data is gold, and now you can mine all of it

At a minute and \$0.32 a run, you can ask questions the way you grep. Anyone on the team can check a hunch against the whole production history, like whether users keep asking for something the agent can't do, before anybody builds anything. Three ways to do it:

### 1. Ask Langy

Ask Langy, the AI engineer inside LangWatch, in chat. It runs the eval on a sample first, then on every conversation, and answers with the count, the conversations and what to change in the agent.

![A Langy chat. The user asks: find me all the conversations where the customer was frustrated. Langy answers with a count out of 10,000 conversations in the last 30 days and three session links](https://langwatch.ai/blog/instant-evals-langy.webp)

*The numbers in this mock are illustrative.*

### 2. Find anything in your data

Run a question over every conversation, trace or single LLM call you have stored. It can be a yes or no ("the agent called an API outside our own systems"), a score, or a category ("what was this conversation about"). `estimate` shows the cost before anything runs:

```bash
langwatch instant-eval estimate "the customer sounds annoyed" \
  --target threads --last 30d --limit 10000
```

![A search bar asking: find me where adding a tool could have helped, over a grid of dots with the matches lit and a match rate under it](https://langwatch.ai/blog/instant-evals-find.webp)

*The numbers in this mock are illustrative.*

Up to ten questions on the same row go to the judge in one request and cost about the same as one, so if you're reading 10,000 conversations anyway, ask everything.

### 3. Export the perfect dataset

The matches are your hard cases: frustrated customers, wrong refunds, failed tool calls. Export them as a test set, a post-training set, or examples to distill a smaller model from.

```bash
langwatch instant-eval results <id> --matched -o json
```

## eval() in LangWatch SQL

LangWatchQL is SQL over everything LangWatch stores, agent traces and coding agent traces included, and `eval()` is now a function in it. The verdict comes back as a column, a probability on every row. This is the statement the CLI writes for a question about conversations:

```sql
SELECT
  argMax(m.TraceId, m.OccurredAt) AS TraceId,
  m.ConversationId AS ThreadId,
  max(m.OccurredAt) AS OccurredAt,
  eval(
    conversation_bounded(m.ConversationId, 8000, ''),
    'the customer sounds frustrated'
  ) AS frustrated
FROM analytics.trace_metrics AS m
WHERE m.OccurredAt >= {start_at:DateTime}
  AND m.OccurredAt < {end_at:DateTime}
  AND m.ConversationId != ''
GROUP BY m.ConversationId
ORDER BY ThreadId
```

![The same SQL statement in an editor window, with the eval call highlighted](https://langwatch.ai/blog/instant-evals-sql.webp)

The CLI prints the statement it ran, so you can copy it and point the `WHERE` at one customer segment, one model version or one week. A run judges up to 100,000 rows. Traces too long for the judge get cut to a digest that keeps the errors and the model calls, and the row is marked as shortened.

## What it costs

\$0.0546 per million input tokens. A product agent conversation of about 1,200 characters comes to about \$0.32 per 10,000 conversations, \$3.20 per 100,000. Full traces are bigger, about 8,400 tokens against 600 for a conversation, so the same 100,000 rows cost about \$46. The estimate tells you which one you're about to run.

## Start today

Instant Evals is available today from the CLI, in LangWatchQL, from Langy in the app and over the REST API. If your traces are already in LangWatch, you can run your first question now. If not, [sign up](https://app.langwatch.ai/) and send them.

More on the [Instant Evals page](https://langwatch.ai/instant-evals) and in the [docs](https://langwatch.ai/docs/api-reference/instant-evals/overview).

## Frequently asked questions

### What are Instant Evals?

Evals you run on demand over data you already stored. You ask a question in plain words, LangWatch judges every conversation, trace or LLM call in the period you pick, and returns the matches, the match rate and a probability on each.

### How are Instant Evals different from online evals?

Online evals score traffic as it comes in, usually a sample of it, against checks you set up in advance. Instant Evals answer a new question over everything you already stored, every row.

### How accurate is it compared to an LLM as a judge?

On a benchmark of 300 support conversations with human labels, Instant Evals agrees with the labels 97% of the time. Claude Opus 5 as a judge agrees 91% of the time, GPT 5 mini 90% and GPT 5.6 88%. Every answer comes with a calibrated probability, so a threshold of 0.8 means what it says.

### What does it cost to evaluate 10,000 conversations?

About $0.32, at $0.0546 per million input tokens. The same 10,000 conversations cost about $31 with Claude Opus 5 as the judge. You see the cost before the run starts.

### How long does it take?

About 1 minute 13 seconds for 10,000 conversations. The LLM judges in the same benchmark take between 10 and 25 minutes.

### Do I need my own model provider keys?

No. LangWatch runs the judge, so there is no provider account to set up and no key to rotate.

### Which model does the judging?

Jev, TypeSafe's new model, a small model made for judging text. We test new judge models as they come out and run your evals on the most accurate one.

### Can I run an eval inside a SQL query?

Yes. eval() is a function in LangWatchQL, the SQL dialect for your LangWatch data. You select the rows with SQL you already know and the verdict comes back as a column.

### Can I ask more than one question at a time?

Yes, up to ten questions per row. They go to the judge in one request, so ten cost about the same as one.

### Can I get the matches out to build a dataset?

Yes. The CLI prints them as a table or as JSON, and the verdicts land in a table you can join to your traces in SQL, so you can build a test set or a fine-tuning set from the hard cases.

### How big can a run be?

Runs go in batches of 100,000 rows, so a million rows is ten runs.