# How to evaluate multi-turn conversations with your AI agent

Multi-turn evaluation grades a whole conversation with your AI agent. Grade it at three levels, the conversation, the trajectory of tool calls and each turn, on conversations a simulated user generates before release and on real threads after.

*By Rogerio Chaves · October 31, 2026*

Canonical: https://langwatch.ai/blog/how-to-evaluate-multi-turn-conversations

![How to evaluate multi-turn conversations with your AI agent](https://langwatch.ai/blog/how-to-evaluate-multi-turn-conversations-cover.webp)

**Short answer:** Multi-turn evaluation grades a whole conversation with your AI agent instead of a single reply. Check it at three levels: the conversation (did the user get what they came for), the trajectory (which tools the agent called across turns) and each turn. Generate conversations with a simulated user before release, and score real threads after.

Your agent can answer every message well on its own and still fail the conversation. It asks for the order number the customer gave two messages ago, or it holds the refund policy on turn one and gives in on turn four. Single-turn evals see neither, because they score one input against one output.

The code below uses [Scenario](https://langwatch.ai/scenario/), the open-source agent testing framework we build at LangWatch, for conversations before release, and LangWatch monitors for the threads in production.

## How do I evaluate multi-turn conversations with my AI agent?

Write down what a good conversation looks like as plain criteria, then check them at three levels: the whole conversation, the trajectory of tool calls across turns, and each turn. Before release you generate the conversations with a simulated user. In production you group each user's messages by thread id and grade the thread.

The criteria matter more than any tool. "The agent does not ask for information the customer already gave" is a check you can run on any conversation, simulated or real, and when it fails you know what to fix. A coherence score of 0.8 doesn't tell you what to fix on Monday morning.

## Why single-turn evals miss the bugs that start on turn two

A single-turn eval runs one input and compares the output with what you expected. The bugs that hurt a chat agent depend on what came before, so a dataset of single messages never triggers them:

1. **Lost context.** The customer gives the order number in the first message, and in the third the agent asks for it again. Each reply is polite and correct on its own.
2. **Contradictions.** Turn two says the order can still be cancelled, and turn five says it shipped yesterday.
3. **Policy drift.** The agent refuses a refund outside the return window, the customer pushes three times, and on turn four it promises one anyway.
4. **Tool misuse mid-conversation.** The customer changes their mind ("actually, just cancel it"), and the agent calls the cancel tool without checking the order status, or calls it twice.

Some teams try to fix this with a dataset of a ticket plus the expected sequence of tool calls. Our [multi-turn cookbook](https://langwatch.ai/docs/cookbooks/evaluating-multi-turn-conversations) explains why that breaks: it assumes one fixed path to the answer, while real conversations bring new information halfway through. The cookbook grades outcomes instead, with criteria like "Agent MUST inform user cancellation is possible IFF order.status != 'shipped'", which hold whatever path the agent took.

## Grade the conversation, the trajectory and each turn

Each level answers a different question, and each needs its own kind of criterion:

| Level | The question | Example criterion | Where you check it |
|---|---|---|---|
| Conversation | Did the user get what they came for? | "The customer ends the chat knowing whether the order can be cancelled" | A judge at the end of a simulation, a Thread Level monitor in production |
| Trajectory | Did the agent do the right things across turns? | "Checks the order status before saying it can be cancelled", "Cancels the order at most once" | A judge that reads the agent's traces, assertions on tool calls |
| Each turn | Was this reply right, given everything before it? | "Does not ask for the order number again" | A judge checkpoint in the middle of a script, a Trace Level monitor in production |

**The conversation level** is where I'd start, because it maps to whether the user got helped. Write criteria about outcomes and actions instead of wording, since the exact words change on every run.

**At the trajectory level** you check what the agent did, which the reply text can't show. An agent can write "your order is cancelled" without ever calling the cancel tool. When the judge reads the agent's traces, one trace per turn, it checks "looks up the order before answering" against the actual tool call span ([linking your traces](https://langwatch.ai/docs/agent-testing/linking-your-traces)). It also catches the conversation that passes for the wrong reason. As I put it on a call in September, when a tool fails and "the agent tries again, and it works at the end, so it's all fine. You actually don't want it to work after trying twice."

**The turn level** is for rules that must hold at one specific moment: never asking for the same detail twice, or staying on the refund policy right after the customer pushes. Keep it to those. A checkpoint on every turn turns into a test that fails whenever the agent phrases something differently.

## Build the conversation with a simulated user

Before release you have no real conversations to grade, so you generate them. A simulated user, an LLM playing the customer, improvises from a short description and keeps the conversation going, so every run tests the behavior on new wording instead of one hand-written transcript.

Here's one scenario that checks all three levels, adapted from the [Scenario docs](https://langwatch.ai/scenario/basics/scripted-simulations/). The script pins the turns where you need a check, and `scenario.proceed()` lets the simulated user improvise in between:

```python
import pytest
import scenario

class SupportAgent(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        return await my_support_agent(input.messages)  # your agent

def check_tool_calls(state: scenario.ScenarioState) -> None:
    # trajectory: across every turn so far
    assert state.has_tool_call("get_order_status")
    assert len(state.tool_calls("cancel_order")) <= 1

@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_cancel_after_address_change():
    result = await scenario.run(
        name="cancel after changing the address",
        description="""
            The customer changed the delivery address of order ORD123 yesterday.
            Now they want to cancel the whole order. They give the order number
            once, in their first message, and get annoyed if asked again.
        """,
        agents=[
            SupportAgent(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(),
        ],
        script=[
            scenario.user("hi, about ORD123, I changed the address yesterday"),
            scenario.agent(),
            scenario.user("actually, just cancel the whole thing"),
            scenario.agent(),
            # each turn: a checkpoint that stops the run if it fails
            scenario.judge(criteria=["Agent does not ask for the order number again"]),
            scenario.proceed(turns=3),
            check_tool_calls,
            # the whole conversation
            scenario.judge(criteria=[
                "Agent checks the order status before saying it can be cancelled",
                "Customer ends the chat knowing whether the order is cancelled",
            ]),
        ],
    )

    assert result.success
```

Each `scenario.judge()` with criteria is a checkpoint: if a criterion fails, the run stops right there, so you know which turn broke. The custom step reads every tool call so far, from the messages and the spans. When your agent runs as a separate service, set `fetch_remote_traces=True` in `scenario.configure()` and the judge reads the agent's own traces before its verdict.

This runs under pytest, and the TypeScript package `@langwatch/scenario` runs the same shape under vitest. If you'd rather not write code, the [scenario editor](https://langwatch.ai/docs/agent-testing/scenarios) in LangWatch takes the same situation and criteria, one per line. Our guide on [how to test AI agents](https://langwatch.ai/blog/how-to-test-ai-agents) covers putting these in CI, and [what AI agent simulations are](https://langwatch.ai/blog/what-are-ai-agent-simulations) goes deeper on the simulated user and the judge.

## Score production threads with monitors

Simulations cover the situations you thought of. To grade the real ones, group each conversation's traces by a thread id and score the threads as they finish.

Every user message creates its own trace, so send the same `thread_id` on each one. In Python it's one call, from the [tracking conversations](https://langwatch.ai/docs/integration/python/tutorials/tracking-conversations) docs:

```python
import langwatch

@langwatch.trace()
def handle_message(thread_id: str, user_id: str, message: str):
    langwatch.get_current_trace().update(metadata={
        "thread_id": thread_id,
        "user_id": user_id,
    })
    # your LLM pipeline logic here...
```

Then add [online evaluations](https://langwatch.ai/docs/evaluations/online-evaluation/overview), which LangWatch calls monitors, at the two levels:

- **Thread Level** for the conversation and the trajectory. An LLM-as-a-Judge reads the full thread, each turn with the agent's tool calls, arguments and results in between, and checks your criteria. It waits until the thread has been idle for five minutes by default, so it doesn't score again on every new message.
- **Trace Level** for single turns, since each trace is one message and its reply. Use it for checks that make sense on one reply, like PII detection or an off-topic answer.

Monitors run after the response is sent, so they never slow down your users. For a new question about your past traffic, [Instant Evals](https://langwatch.ai/docs/features/instant-evals/overview) judges every conversation in a time window instead of a sample, at about a minute per 10,000 conversations:

```bash
langwatch instant-eval run "the agent asked for something the customer had already given" --target threads --last 30d
```

Then close the loop. An automation can add every trace a monitor failed to a [dataset](https://langwatch.ai/docs/datasets/overview), and from an open trace you can add the whole conversation as one row. Each of those conversations is a candidate for a new scenario, so the bug you found in production gets a test that runs before the next release.

If you have no criteria yet, read twenty real threads and write down what went wrong in each one. Those notes are your first criteria.

## Frequently asked questions

### What is multi-turn evaluation?

Multi-turn evaluation grades a conversation between a user and an AI agent across all its turns, instead of one input and one output. It catches the bugs that depend on earlier messages: the agent forgetting what the user said, contradicting itself, giving in on a policy after the user pushes, or calling a tool at the wrong moment.

### What should I measure in a multi-turn conversation?

Write criteria about outcomes and actions, in plain language: the user got what they came for, the agent checked the order status before promising a cancellation, the agent never asked for information the user already gave. A judge passes or fails each one. Generic scores like coherence are hard to act on, while a failed criterion tells you what to fix.

### Can I evaluate multi-turn conversations without a dataset of conversations?

Yes. Use a simulated user: an LLM plays the user from a short description of their situation and talks to your agent for several turns, and a judge grades the result. Scenario, the open-source framework from LangWatch, does this inside pytest or vitest. Once you have production traffic, add real conversations to a dataset, one row per thread.

### How do I group production messages into conversations?

Send the same thread_id in the metadata of every trace that belongs to one conversation, usually your chat or session id. LangWatch then groups those traces into one thread, which you can open as a whole, score with a Thread Level monitor and add to a dataset as one row.

### Should I score every turn or the whole conversation?

Both, for different questions. Score the whole conversation for the outcome, the tool calls across turns for what the agent did, and single turns only for rules that must hold at one moment, like never asking for the same detail twice. Start with the conversation level, since it maps to whether the user got helped.