LangWatch Instant Evals
Search your entire production history
Type a question in plain words. Instant Evals judge every conversation, trace or LLM call your agent produced in the period you pick, and return the matches, each with a probability.
Eval accuracy above frontier models, at a fraction of the time and cost
Instant Evals agrees with the human labels more often than Claude Opus 5 and GPT 5.6, at a hundredth of the cost.
Cost to judge 10,000 conversations
Lower is betterTime to judge 10,000 conversations
Lower is betterEach judge ran at the highest concurrency its provider allowed; cost at each provider's list price. Agreement with the human labels: Instant Evals 97%, Claude Opus 5 91%, GPT 5 mini 90%, GPT 5.6 88%.
An Eval inside your SQL query
LangWatchQL lets you filter, group and aggregate your traces any way you want, and now it runs evals inside the same query.
- 1
Extract any data, in any shape
Conversations, traces or single LLM calls, selected with SQL you already know.
- 2
Get an intelligent verdict on every row
eval() judges each one, up to 100,000 in a run.
- 3
Use it as a dataset right away
Export the matches as JSONL for post‑training, prompt tuning or a test set.
SELECT
argMax(m.TraceId, m.OccurredAt) AS TraceId,
m.ConversationId AS ThreadId,
max(m.OccurredAt) AS OccurredAt,
eval(conversation(m.ConversationId), 'the customer sounds frustrated') AS frustrated
FROM analytics.trace_metrics AS m
WHERE m.OccurredAt >= {start_at:DateTime}
AND m.OccurredAt < {end_at:DateTime}
AND m.ConversationId != ''
GROUP BY m.ConversationId
ORDER BY ThreadIdYour production data is gold, and now you can mine all of it
Every conversation your agent ever had is stored already. Ask it anything, judge every row, and turn the matches into prompt fixes, fine‑tuning sets and the next release.
Ask Langy
Langy, the AI engineer inside LangWatch, does the digging for you: ask in chat, and it comes back with the count, the conversations and what to change in the agent.
Meet LangyFind anything in your data
Ask a question in plain words over every conversation, trace or LLM call you have stored, and get back the matches, the match rate, and a probability on each.
Read the docsExport the perfect dataset
Export the matches as JSONL and use them as a test set, a post‑training set, or the examples to distill a smaller model from.
Export from the CLIAsk every question you have
Ask up to ten questions per row for the cost of one. You see the cost before the run starts.
Each row is about 591 tokens: the conversation plus 300 for the question. Extra questions add 300 tokens each, the conversation is sent once.
Questions
What about online evals?
Online evals score traffic as it comes in, usually a sample of it. Instant Evals answer a new question over everything you already stored, every row, and give you the matches and the rate.
What can it judge?
A whole conversation, one trace, or a single LLM call. A question can ask for yes or no, a score, or a category. Up to ten questions on the same row go to the judge in one request, so ten cost about the same as one.
How accurate is it?
On the 300 conversations in the benchmark above, Instant Evals agrees with the human labels 97% of the time. Claude Opus 5 as a judge agrees 91% of the time, GPT 5 mini 90%, GPT 5.6 88%. Every answer comes with a calibrated probability, so a threshold of 0.8 means what it says.
What does it cost?
About $0.32 for 10,000 conversations, at $0.0546 per million input tokens. You see the cost before the run starts, and the estimate command shows it without running anything.
Do I need my own model provider keys?
No. LangWatch runs the judge, so there is no provider account to set up and no key to rotate.
Can I get the matches out?
Yes. The CLI prints them and exports them as JSONL. The verdicts also land in a table you can join to your traces in SQL, so you can build a dataset from the hard cases.
How big can a run be?
Runs go in batches of 100,000 rows, so a million rows is ten runs. A trace too long for the judge is cut to a digest that keeps the errors and model calls, and the row is marked as shortened.
Where do I run it?
From the CLI, as a function inside a SQL query in LangWatch, from Langy in the app, or over the REST API.
Ask your production history a question.
Works from the CLI, SQL or Langy. Read the docs