# How to test AI agents: a step-by-step guide

Agents break between turns, and you can't retest twenty chats by hand after every prompt change. Test them with a simulated user, a judge and CI, then feed production failures back in.

*By Rogerio Chaves · October 20, 2026*

Canonical: https://langwatch.ai/blog/how-to-test-ai-agents

![How to test AI agents: a step-by-step guide](https://langwatch.ai/blog/how-to-test-ai-agents-cover.webp)

**Short answer:** To test an AI agent, write scenarios from real user situations, let a simulated user talk to your agent, and have a judge pass or fail the conversation against your criteria. Run them in CI on every change, and turn each production failure into a new scenario.

The code below uses [Scenario](https://langwatch.ai/scenario/), the open-source agent testing framework we build at LangWatch. On calls I describe it as Playwright for agents: you write tests for your agent the way you write end-to-end tests for your frontend.

## How can I test my AI agents?

Test them in three layers: unit tests for the deterministic code, evals for single steps like retrieval, and simulations for the agent end to end. Simulations are the part people skip, and it's where agent bugs live, because they only show up over a few turns of conversation.

We call this the [agent testing pyramid](https://langwatch.ai/scenario/best-practices/the-agent-testing-pyramid/), and the [agent testing overview](https://langwatch.ai/docs/agent-testing/overview) in our docs covers the top layer:

| Layer | What it tests | Example | How many |
|---|---|---|---|
| Unit tests | The plain code around the model: tool wrappers, parsing, auth, rate limits | "The order lookup tool returns the right order for an id" | Many |
| Evals | One probabilistic step, scored on a dataset | "Retrieval finds the refund policy for 95 of 100 questions" | Hundreds of rows |
| Simulations | The whole agent over a multi-turn conversation | "A customer asks for a refund on a year-old order, and the agent looks it up and doesn't promise the refund" | Tens of scenarios |

A button you tested last week keeps working. An agent can break tomorrow because someone changed one line of the prompt or the provider moved a model version, however smart the model gets. So the top layer has to run on every change, the same as your end-to-end tests do.

## Why unit tests and a few manual chats miss agent bugs

Unit tests check functions, and agent bugs happen between turns: the agent forgets what the user said two messages ago, calls the wrong tool, or fixes one behavior and breaks another. Manual chats catch some of it, but you won't retest twenty conversations by hand after every prompt change.

The AI team at a large DIY retail chain learned this on their home improvement assistant. One of their engineers described the start on a call in July: "in the beginning, I was retesting, like, software. So you, try to test feature, but you cannot really do that."

Testing one behavior at a time went badly for them. Their assistant kept fixating on things like lead paint, and it tried to send customers with ordinary gas questions to the emergency number. Every fix for one of those broke something else, and another engineer on the team called it "a game of whack-a-mole". What got them out was a balanced set of scenarios that runs as a whole on every change, so a fix that breaks a neighboring behavior fails right away.

Manual chatting still matters, it's how you notice an odd answer like those lead paint warnings in the first place. The problem is that it doesn't scale: a bug you found by chatting and never wrote down can come back with the next prompt change.

## Step 1: write the scenarios your users actually bring

A scenario is one situation a real user brings, plus the criteria the agent must meet. Start with five to ten: the main jobs your agent does, and the mistakes that would cost you money or trust, like promising a refund you don't give.

Write the situation the way a user would live it, with their context and mood, and keep the criteria concrete. Here's the example from our docs: a customer asks for a refund on an order from last year. The criteria are that the agent must look up the order and must not promise the refund.

Include "must not" criteria on purpose. The failures that hurt are usually the agent doing something it shouldn't, like a refund it can't give or advice it shouldn't offer, and those are the ones that end up in your support queue.

The usual plan for this step is "later". An engineer on a team shipping in two-week sprints put it in one line on a call this October (translated from Portuguese): "Then we think about optimizations and then we think about evals." My advice was neither before nor after the feature, but during it. Ask Claude Code to write the scenario while it builds the feature. The [scenarios skill](https://langwatch.ai/docs/skills/directory) (`npx skills add langwatch/skills/scenarios`) teaches your coding agent how, and the test can fail at first and still tell you what's missing.

## Step 2: let a simulated user hold the conversation

Instead of scripting every user message, describe the user and let a simulated user, an LLM playing that person, talk to your agent. It uses different wording on every run, so you test the behavior instead of one lucky phrasing.

In Python, install `langwatch-scenario` and wrap your agent in an adapter. This runs under pytest:

```python
import pytest
import scenario

class SupportAgent(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        return await my_support_agent(input.messages)

@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_refund_on_old_order():
    result = await scenario.run(
        name="refund on an order from last year",
        description="""
            The customer bought headphones fourteen months ago and they broke.
            They want their money back. Polite, but they won't give up easily.
        """,
        agents=[
            SupportAgent(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(criteria=[
                "Agent looks up the order before answering",
                "Agent does not promise a refund",
                "Agent offers the customer a next step",
            ]),
        ],
    )

    assert result.success
```

`scenario.configure(default_model=...)` sets the model the simulated user and the judge run on. In TypeScript the package is `@langwatch/scenario`, it runs under vitest, and the shape is the same with `scenario.userSimulatorAgent()` and `scenario.judgeAgent()`. The adapter can call your agent in-process or over HTTP, so it works with LangGraph, CrewAI, Pydantic AI, Mastra, the Vercel AI SDK or your own code ([scenarios in code](https://langwatch.ai/docs/agent-testing/scenarios-in-code) has both languages in full).

When a test needs a fixed opening line or a check in the middle, add a script. The rest of the conversation still plays out on its own:

```python
def check_order_lookup(state: scenario.ScenarioState):
    assert state.has_tool_call("lookup_order")

# passed to scenario.run(..., script=[...])
script=[
    scenario.user("hi, my headphones broke and I want my money back"),
    scenario.agent(),
    check_order_lookup,
    scenario.user(),
    scenario.agent(),
    scenario.judge(criteria=["Agent does not promise a refund"]),
]
```

## Step 3: judge the whole conversation against criteria

A judge agent reads the full conversation after each agent turn and decides to continue, pass or fail, with its reasoning for each criterion. You get a yes or no on "can the agent do this job", which a PM can read too.

Write criteria the way you would explain the job to a new support hire: "asks for the order number before checking the status", "escalates when the customer asks for a human". One scenario should test one situation. If you find yourself writing hundreds of tiny variations of the same step, like picking the right language or the right route, that belongs in a dataset eval instead.

The judge can also look past the text. When your agent runs as a separate service, turn on `fetch_remote_traces=True` and the judge reads the traces your agent sends to LangWatch, so a criterion like "the agent wrote the ticket to the CRM" gets checked against the CRM call in the trace. For checks a library already does well, like faithfulness on a RAG answer or comparing against a golden answer, you can [attach evaluators](https://langwatch.ai/docs/agent-testing/evaluators) that run after the judge and can fail the scenario too.

Before you trust a new scenario, watch it fail. Break the behavior on purpose, or write the scenario for a bug before you fix it, and check that the judge catches it, because a test you've only ever seen pass could be passing for the wrong reason.

## Step 4: run the tests in CI on every change

Run the scenarios in CI on every pull request, the same way you run your other tests. Prompt edits, model updates and tool refactors all change agent behavior, and CI is where you find out before your users do.

Scenarios written in code are plain pytest or vitest tests, so your existing workflow runs them. Add your keys as secrets and set a batch id, so every run from one CI job groups together in LangWatch:

```yaml
- name: Run Scenario tests
  run: uv run pytest tests/ -v
  env:
    LANGWATCH_API_KEY: ${{ secrets.LANGWATCH_API_KEY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    SCENARIO_BATCH_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
```

If your PMs or domain experts write scenarios in the LangWatch platform instead of in code, CI starts that test suite with one command, and the job fails when a run fails ([run from CI](https://langwatch.ai/docs/agent-testing/run-from-ci)):

```bash
langwatch test-suite run "Checkout regression" --target http:agent_abc123 --wait
```

The same suite answers whether you can switch models. Your prompts, memories and tool descriptions slowly get tuned to the model you built with, so a newer model can do worse on your agent even when it's better in general. With the suite in place, you swap the model, read the pass rate and decide.

## Step 5: turn production failures into new tests

Simulations only cover the situations someone thought to write down, so trace your agent too. Every time a real conversation goes wrong, turn that trace into a scenario that reproduces it, fix the agent, and keep the scenario in the suite so the bug can't come back.

A founder building an accounting agent asked exactly this on a call in July. His Claude Code already dug the bad cases out of his production database, and the missing step was: "how do we fix them, and then how do we test them for, like, not breaking it again?"

With your agent traced in LangWatch, the loop goes like this (we call it the [vibe-eval loop](https://langwatch.ai/scenario/best-practices/the-vibe-eval-loop/)):

1. Find the bad conversation and copy its trace ID.
2. Give it to your coding agent with the LangWatch skills installed. It reads the trace with `langwatch trace get <trace-id>` and writes a scenario that reproduces the problem, with the external tools [mocked](https://langwatch.ai/scenario/testing-guides/mocks/) so it runs offline.
3. Run it and watch it fail for the same reason the real conversation did.
4. Fix the prompt, the tool or the code, and watch it pass.
5. Commit the scenario. From now on CI runs it on every change.

If you're starting from zero, the last conversation a user complained about makes a fine scenario number one.

## Frequently asked questions

### Which library can I use to write tests for my AI agent?

Scenario is an open-source agent testing framework from LangWatch, in Python (langwatch-scenario) and TypeScript (@langwatch/scenario). It runs inside pytest or vitest: a simulated user talks to your agent, and a judge passes or fails the conversation against criteria you write in plain language. It works with any agent framework, including LangGraph, CrewAI, Pydantic AI, Mastra and the Vercel AI SDK.

### How do I write tests for my AI agent?

Describe one situation a real user brings, list what the agent must and must not do, and run it as a scenario: a simulated user holds the conversation and a judge checks each criterion. Add assertions on tool calls where you need a hard check. Run the scenarios in CI so every prompt, model or tool change gets tested.

### What is the difference between agent testing and evals?

An eval scores one input against an expected output on a dataset, which fits one step inside the agent, like retrieval or a single prompt. An agent test scores a whole multi-turn conversation, including the tools the agent called. Use evals for the parts and agent tests for the agent end to end.

### Can a simulated user really behave like my users?

Not perfectly, and no simulator does. It varies its wording on every run and follows the situation you describe, which catches a lot before release. The cases you never imagined show up in production, so trace your agent and turn each real failure into a new scenario.

### Do I need a LangWatch account to use Scenario?

No. Scenario is open source and runs in your own test runner. A LangWatch API key is optional: with it, every run shows up in LangWatch with the full conversation, the judge's reasoning and the pass rate over time.