Instant Evals: Search Your Entire Production History

What You Can Do Now

Instant Evals

Every conversation your agent ever had is already stored in LangWatch. Until now, asking a question of all of it meant reading a sample by hand or paying a frontier model to read every row. Instant Evals run a judge built for exactly this job, so the answer over your whole history comes back in minutes instead of a sample.

langwatch instant-eval run "the customer sounds frustrated" --target threads --last 30d

In the run above, the judge read 10,000 conversations from the last 30 days in 2 minutes 26 seconds, found 3,351 where the customer sounded frustrated, and charged $0.62 for the 11.4 million tokens it read.

More accurate than a frontier model as a judge

On a benchmark of 300 support conversations labelled by hand, Instant Evals agreed with the human labels 97% of the time. Claude Opus 5 as a judge agreed 91% of the time, GPT 5 mini 90% and GPT 5.6 88%. The same 300 conversations cost $0.02 and took 2.1 seconds with Instant Evals, against$2.30 and 45.5 seconds with Claude Opus 5, with each judge at the highest concurrency its provider allowed.

Every verdict carries a calibrated probability, so a threshold of 0.8 means what it says.

Ask it anything, in the shape you need

• Judge a whole conversation (--target threads), a single trace (traces) or a single LLM call (llm_spans), and narrow the rows first with the same filter the traces page uses.
• Ask for yes or no, a score, or a category. Up to ten questions on the same row go to the judge in one request, so ten questions cost about the same as one.
• run waits and prints the answer. sample shows exactly what the judge read, results --matched pages through the rest, and the matches export as JSONL for a test set, a fine-tuning set or the next prompt fix.
• Every run is one LangWatchQL statement with eval() inside it, and the CLI prints that statement, so when the shorthand runs out you edit the query instead of starting over. eval_score, eval_category, eval_passed and eval_criteria cover the other kinds of question.
• Judge up to 100,000 rows in one run on a paid plan.

Priced before it runs

• --estimate prices a run and exits without judging or charging anything, and a run over a thousand rows shows you the estimate before it starts.
• $0.0546 per million input tokens, and output is free. Organizations without a paid plan start with$1 of Instant Evals.
• LangWatch runs the judge, so there is no provider key to set up. Project budgets on the LangWatch AI Gateway cap Instant Evals like any other spend.

Start with the Instant Evals docs or the CLI reference.

LangWatchQL from the terminal and from your agent

• langwatch query runs a LangWatchQL statement from the CLI, and the MCP server's run_query tool does the same for a coding agent.
• POST /api/v1/query takes any LangWatch API key and reads across every project that key can read, over 47 views. Captured text and cost columns follow each project's privacy settings, and a caller without access is refused rather than handed a quietly stripped row.
• GET /api/v1/query/reference describes LangWatchQL and the trace filter language in one payload, with a table saying which of the two answers which kind of question.
• The traces API accepts the same filter and facets the traces page uses.

Notes

• Evaluators: a Claude judge could fail with "temperature and top_p cannot both be specified", because two dispatch paths handled generation parameters differently. Both paths now send the same configuration and temperature wins for Claude. Ragas evaluators stop skipping entries above 16,384 tokens, and skipped rows show as skipped.
• Evaluator spans no longer turn into orphan traces when the evaluated trace has no parent link.
• Sign-in: a Google, GitHub or Microsoft button no longer lets a member into an organization that enforces single sign-on. They are sent to their own identity provider instead.
• Automations: trigger filters on metadata.value resolve bare OpenTelemetry attribute names.
• Coding agents: session usage is charged to the branch declared under it, not across the session's whole lifetime.
• Model providers: whitespace around a pasted credential is stripped before it is stored.