# AI evals: how to know your agent actually does the job

AI evals are tests for your AI agent: a real case, a run, a grader and a score you compare between versions. This guide follows one returns agent from a single conversation to evals that run on every release, with the LangWatch screens for each step.

*By Rogerio Chaves · November 8, 2026*

Canonical: https://langwatch.ai/blog/what-are-ai-evals

![AI evals: how to know your agent actually does the job](https://langwatch.ai/blog/what-are-ai-evals-cover.webp)

An AI eval is a test for your AI agent: you give it a real case, let it run, and a grader checks whether it did the job, so you get a number instead of a feeling. If your agent looks fine when you try a few questions by hand before a release, evals tell you whether it holds up on the thousand conversations you never read.

I'll use one small agent the whole way through. It handles returns for ACME, a made-up shop. I gave it one case: the customer's blender from order 1001 arrived broken, and they want their money back. The agent has a tool, `check_return_eligibility`, that only accepts a reason from a fixed list, like `defective`, and the tool's description never says so.

I ran that conversation six times. In four of them the agent tried `arrived broken`, then `damaged`, gave up, and told the customer: "Please contact ACME support so we can handle this manually." The agent is ACME support. Then I asked a judge model to grade the replies, and it rated all six polite and helpful.

When a language model fails, it writes a friendly paragraph, so trying a few questions by hand tells you how your agent sounds and very little about whether it worked.

## If you read nothing else

An AI agent can write a polite reply while failing the job, so an eval checks what the agent did, and gives you a number you can compare between versions.

- An eval has four parts: a case, a run of your agent, a grader that says pass or fail, and a score you compare with the last version. It's a unit test for software that can answer the same question differently every time.
- Grade what the agent did as well as what it said: the tool calls, the steps, and whether the customer got their refund.
- You don't need a big dataset to start. You learn more from ten cases you can read than from 10,000 you can't, and your production traces already hold the first ones.
- The grader needs testing too. A judge that disagrees with your domain experts gives you a confident wrong number.
- Run evals before every change, on every live conversation, and again whenever you switch models.
- If you lead the team, ask for two numbers: how often the agent does the job, and how often the grader agrees with your experts.

## What is an AI eval

An AI eval is a repeatable check of your agent on one case, with a grader that decides pass or fail. Run it over many cases and you get a pass rate you can compare between versions.

For the returns agent, the four parts look like this:

1. **The case.** The situation and what a good result is. "The blender from order 1001 arrived broken, I want a refund." A good result is a created return and a refund to the card the customer paid with.
2. **The run.** The agent handles the case with its real tools, and you keep the trace: every model call and tool call, with what went in and what came out.
3. **The grader.** Something that says pass or fail. It can be a few lines of code, a judge model answering a yes-or-no question, or a person.
4. **The score.** The pass rate over all your cases, next to the pass rate of the last version.

An eval is different from a benchmark. A benchmark compares models on public tasks someone else wrote, and an eval checks your agent on your own customers' cases, so a model that tops a benchmark can still fail your returns flow.

## What should an eval check

For an agent, three things: whether the user got what they came for, whether the steps made sense, and whether each tool call worked. Read only the reply and you check the first one, badly.

The four bad runs of the returns agent passed on the reply, because "please contact ACME support" sounds helpful. They failed all three checks once you read the trace: no return was created, and the agent called `check_return_eligibility` twice with reasons it doesn't accept, so both calls failed.

You read the trace in **Trace Explorer**. Click a trace, pick the **Trace** tab, and the **Waterfall** view shows every span with its timing, input and output.

![A trace in LangWatch on the Trace tab, with the Waterfall view: each span of the turn with its duration, model and cost](https://langwatch.ai/blog/framework-for-evaluating-agents-trace-waterfall.webp)

Our post on [AI agent evaluation](https://langwatch.ai/blog/framework-for-evaluating-agents) goes through this trace span by span.

## Where do the first test cases come from

From what you already have: your documents if you have no users yet, and your traces if you do.

1. **No users yet.** Your agent is in development or a closed pilot. Your cases are in what you wrote down: the spec, the help center, the refund policy, the support tickets your team answered before the agent existed, and every prompt fix in your git history, because each one was a bug somebody found.
2. **Real users already.** Your cases are in their conversations. Search your traces for the ones that went wrong, select them and click **Add to dataset**.

![The Add to Dataset drawer with input mapped to input and output mapped to expected_output, and a preview of the rows that will be added](https://langwatch.ai/blog/build-an-evaluation-dataset-from-production-traces-add-to-dataset-drawer.webp)

When teams ask me how much data they need, I tell them we don't want 10,000 examples, we want like 10 scenarios. That's plenty to start, because you can read all ten and see why each one failed, while a big generated set gives you a percentage and little else.

A dataset written before launch only holds the cases you imagined, so don't spend weeks polishing it. I'd run a first version this week and let the failures tell you what to add. [How to start evals when you have no dataset](https://langwatch.ai/blog/how-to-start-evals-when-you-have-no-dataset) walks through both paths, and [build an evaluation dataset from production traces](https://langwatch.ai/blog/build-an-evaluation-dataset-from-production-traces) covers the second one click by click.

## Why read traces before you write a grader

You can only write a check for a failure you've seen. I didn't know the returns agent gives up until I read the traces, and no grader I'd have written in advance would have asked about it.

So read twenty traces first, and write down what went wrong in your own words. Then count how often each failure happens across all your traffic. In LangWatch you can ask it as a question in the Trace Explorer search bar, with an [Instant Eval](https://langwatch.ai/docs/features/instant-evals/overview):

```text
eval.trace:"the agent sent the customer to support instead of finishing the return"
```

![The Trace Explorer with an Instant Eval chip in the search bar and the matching traces in the table](https://langwatch.ai/blog/framework-for-evaluating-agents-eval-chip.webp)

It judges every trace in the time window, about 10,000 conversations in a minute for \$0.32, so the one conversation in five hundred where the agent gives up is in the result. A judge you pay per call usually reads a sample, and a sample misses that one.

## How do you grade an AI agent

Use code when the check is exact, and a judge model when the check needs reading.

Code checks come first, because they're free, instant and never change their mind. "Did the agent call `create_return`?" and "how many tool calls failed?" are code checks, and with LangWatch SQL you count them straight from the spans, with no model involved:

```sql
SELECT
  s.TraceId AS trace_id,
  countIf(s.SpanAttributes['langwatch.span.type'] = 'tool' AND s.StatusCode = 2) AS failed_tool_calls,
  countIf(s.SpanName = 'create_return') AS returns_created
FROM langwatch.spans AS s
WHERE s.StartTime >= subtractDays(now(), 1)
GROUP BY trace_id
ORDER BY failed_tool_calls DESC
LIMIT 50
```

A judge model reads the conversation and answers a question about it. It catches what code can't, like a promise the policy doesn't allow, but only if you ask the right question and show it the right thing. I asked the returns traces two questions, once about the reply and once about the work:

| What happened | "The reply is polite and helpful" | "The agent created the return the customer asked for" |
|---|---|---|
| Gave up, sent the customer to ACME support | 0.90 | 0.03 |
| Got there on the third try | 0.75 | 0.94 |
| Fixed tool, right the first time | 0.73 | 0.94 |

Each number is the judge's probability that the statement is true. The run that left the customer with nothing got the best score on the reply question, and the question about the work split the runs cleanly.

So I write each judge question about one thing, as a yes or no, about the outcome I want, and I give the judge the whole trace, with the tool calls and their results, instead of only the reply. In a LangWatch online evaluation, the judge's input is **Full Trace (AI-Readable)** by default, which is every span with its tool calls, arguments and results.

## Who checks the grader

Your domain experts check it, by scoring a sample of the same traces the judge graded. A judge is a model too, so it can be wrong in the same confident way your agent is, and a pass rate from a wrong judge is worse than no number, because people trust it.

Those are the people who know what a good answer is: the support lead for the returns agent, a nurse for a health assistant. In LangWatch you select traces and click **Add to annotation queue**, or set up an [automation](https://langwatch.ai/docs/features/automations) that fills the queue for you, and each expert works through the queue one trace at a time.

![An annotation queue item in LangWatch: the conversation, the annotation composer next to it and the bar with Previous, Next and Add to dataset](https://langwatch.ai/blog/what-are-ai-evals-annotation-queue.webp)

Their scores and the judge's verdicts both live in LangWatch, so you can put them side by side in a LangWatch SQL query and count where they disagree. There will be some. Every disagreement is either a judge question to rewrite or a case the judge can't handle. Do it again whenever you change the judge model or your traffic changes, because new kinds of conversations bring cases the judge never saw when you checked it.

The judge model matters too. On our [LLM-as-a-Judge Pareto Frontier](https://langwatch.ai/benchmarks/llm-as-a-judge) benchmark, frontier models were up to 22 points more accurate than Jev, TypeSafe's new judge model, and cost over 100 times more: about \$123 per 1,000 calls for Claude Opus 5.5 against \$0.49. Both have a job. A small judge is the one to search every trace for a pattern, and a frontier judge is the one to put on a release gate, where every point counts.

## How many test cases do you need

Enough that one case doesn't swing the score. With 20 cases, each case is 5 points, so a jump from 70% to 75% can be one lucky row.

So compare the old and the new version on the same cases, run each case more than once, because the agent answers differently every time, and open every row that flipped instead of trusting the average. In an [experiment](https://langwatch.ai/docs/evaluations/experiments/overview), tick two runs under **Results** to see their cost, latency and pass rate side by side, then check row by row that the fix didn't break what passed before.

![Two experiment runs compared side by side in LangWatch: Total Cost, Avg Latency and the Answer correctness pass rate of each](https://langwatch.ai/blog/framework-for-evaluating-agents-compare-runs.webp)

The fix for the returns agent was one line: I put the accepted reasons into the tool's schema as an enum. The old version failed twelve tool calls over six runs and created two returns. The new one failed none and created all six, so every customer got their refund on the first try.

The single number is also what your leadership will ask for. Altura, who rebuilt their AI quality process with LangWatch, put it this way: "Leadership just wants to hear 'we went from 70% to 80%.' But inside that 70%, there's so much compounding going on."

## How do you test an agent that talks in turns

With simulations, where a simulated user talks to your agent for several turns and a judge checks your criteria against the agent's traces at every turn.

A returns conversation is rarely one message. The customer forgets the order number, changes their mind about the refund method, or gets annoyed when the agent asks twice. So you write the situation and the criteria in a scenario, in **Agent Testing** or with [Scenario](https://github.com/langwatch/scenario), our open-source testing library:

```text
The agent looks up the order before it answers
The agent calls check_return_eligibility once, with one of the accepted reasons
The agent creates the return and confirms the refund method
```

![A failed simulation run in LangWatch: a simulated user tries to stack two discount codes, and the judge explains why each of the three criteria failed, checking the transcript against the traces](https://langwatch.ai/blog/framework-for-evaluating-agents-judge-reasoning.webp)

Make a tool fail on purpose in one scenario, and check that the agent recovers instead of handing the job back to the customer. Freeday, whose AI agents handle customer service for enterprise companies, cut onboarding a new customer from half a day to around 10 minutes once their scenarios ran across the team.

I write the criteria about the outcome and about what must never happen, like "creates exactly one return" or "never promises a refund before checking eligibility", and leave the exact order of the calls out on purpose.

A team I was helping moved their agent to a newer OpenAI model, which started to double-check more and ask the user more questions, and their tests failed it for doing the right thing, because the tests expected the same path every time.

[How to test AI agents](https://langwatch.ai/blog/how-to-test-ai-agents) has the scenario code and how to run it in CI.

## How do you evaluate an agent in production

A judge on live traffic catches the cases your tests never thought of, and an automation sends each one to someone who'll look at it. Your users will always bring cases you didn't write.

Open **Online Evals**, click **New Online Evaluation**, and pick **Trace Level**, or **Thread Level** to judge whole conversations. The judge runs after the response is sent, so it never slows your users down, and its verdict and reason show up on every trace.

![The New Online Evaluation drawer in LangWatch with Trace Level and Thread Level, the selected evaluator and its name](https://langwatch.ai/blog/framework-for-evaluating-agents-online-evaluation.webp)

Then click **Automate** on a filtered search, and LangWatch sends each new matching trace to Slack, an annotation queue or a dataset as it arrives. That's the loop: a failure in production becomes a row in a dataset, then a scenario, and it can't ship again without a test failing.

## How to make evals a habit

Run evals on every commit, on every change to a prompt, a model or a tool, and all the time on live traffic.

1. **Every commit.** Code checks and scenarios run in CI, like unit tests.
2. **Every change to a prompt, a model or a tool.** An experiment compares the old and the new version on the same cases before you switch.
3. **All the time.** Online evals judge live traffic, and when you have a hunch, an Instant Eval checks it against every trace while you're still looking at it.

And check the grader again with your experts every time you change the judge or the traffic changes.

## What to ask your AI team

If you sign off on an AI feature, ask your team how often the agent does the job and how often the grader agrees with your experts. These are the questions I'd bring to the next release review:

- How often does the agent do the job, on which cases, and how does that compare to the last release?
- How often does our grader agree with our experts, and when did we last check?
- What broke in production since the last release, and is each one a test now?
- What does one successful conversation cost? [Cost per successful task](https://langwatch.ai/blog/cost-per-successful-task) explains why that's a better number than cost per call.

If your team can't answer the first one yet, the last ten conversations your agent had are where to start.

## Frequently asked questions

### What are AI evals?

AI evals are tests for an AI agent or LLM app. Each one gives the agent a real case, lets it run, and has a grader decide whether it did the job, so you get a pass rate you can compare between versions instead of a feeling from trying a few questions by hand.

### What is the difference between an eval and a benchmark?

A benchmark compares models on public tasks someone else wrote. An eval checks your own agent on your own customers' cases, so a model that tops a benchmark can still fail your eval.

### Do I need a big dataset to start evals?

No. Ten cases you can read are enough to start. With no users yet, write them from your spec, help center, policies and old support tickets. With real users, take them from the conversations in your traces that went wrong.

### Should I grade AI evals with code or with an LLM judge?

Use code when the check is exact, like whether a tool call failed or a return was created, because it is free and never changes its mind. Use an LLM judge with a yes-or-no question when the check needs reading, give it the whole trace, and check its verdicts against your domain experts.

### How do I evaluate an AI agent in production?

Run a judge on live traffic with online evaluations, which run after the response is sent, and send the conversations it flags to Slack, an annotation queue or a dataset. Each failure you find becomes a test case, so it can't ship again unnoticed.