LangWatch tests AI agents with simulated users. A simulated user talks to your agent over a multi-turn conversation, then a judge reads the conversation and the agent's traces and passes or fails each of your criteria, with its reasoning. Write scenarios in the LangWatch platform with no code, or in code with Scenario, LangWatch's open-source testing library, as pytest or vitest tests that run in CI.
Quick setup? Paste the Connect your agent prompt into your coding agent to connect your agent and run a first test suite, or the scenarios prompt to add code-first scenario tests.
What agent testing is for
Agent testing ensures your agent handles real conversations correctly before they reach production. You describe a situation and the criteria the agent must meet, and LangWatch runs a simulated user through the conversation, then a judge gives a pass or fail for each criterion with its reasoning.
For example, a customer asks for a refund on an order from last year. The criteria: the agent must look up the order and must not promise the refund. The simulated user plays the customer with different words on every run, and the judge checks each criterion against the conversation and the agent's own traces.
Use it to:
- Check the agent before a release. Run the test suite and read the pass rate.
- Turn a production incident into a permanent test, so the same mistake does not come back.
- Catch a regression when a prompt, a model or a tool changes.
How a test runs
| Part | What it does |
|---|---|
| Simulated user | Plays the user in the situation the scenario describes, with different wording on every run. |
| Your agent | Answers each turn as it does in production. |
| Judge | Reads the conversation and the agent's traces, then passes or fails each criterion and explains why. |
A scenario is one situation and its criteria. A test suite groups the scenarios you run together. A run plan is the history of one configuration, and the Results tab shows its pass rate over time.
Two ways to run
Both use the same simulated user, the same judge and the same results pages. The difference is where you write the scenarios.
New to writing scenarios in code? Test AI agents with simulated users in pytest or vitest builds a first test step by step.
What you can test
Agent testing vs evaluations
An evaluation scores one input against one expected output, on a dataset. It fits one step inside the agent, such as a retrieval step or one prompt, where you optimize a metric on many examples.
For example, an evaluation dataset has rows like these:
| query | expected_answer |
|---|---|
| What is your refund policy? | We offer a 30-day money-back guarantee on all purchases. |
| How do I cancel my subscription? | You can cancel your subscription by logging into your account and clicking the "Cancel Subscription" button. |
An agent test scores a whole conversation. The scenario describes the user's situation, and the criteria check each step, including the tools the agent called:
script=[ scenario.user("hey I have a problem with my order"), scenario.agent(), expect_ticket_created(), expect_ticket_label("ecommerce"), scenario.user("i want a refund!"), scenario.agent(), expect_tool_call("search_policy"), scenario.user("this is ridiculous! let me talk to a human being"), scenario.agent(), expect_tool_call("escalate_to_human"),]Use evaluations for the parts inside the agent. Use agent testing to check the agent end to end: before a release, to reproduce an incident, and to catch a regression. Both run in the same LangWatch project.
Scenario is the most advanced agent testing framework available. The platform and the SDKs both run on it: the same simulated user and the same judge test text, voice and adversarial conversations, in Python, TypeScript and Go, against any agent framework.
Go deeper
Further reading
- What are AI agent simulations? explains agent simulation testing, how it differs from multi-agent research, and why scenarios should start from production conversations.