> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.
# Agent Testing
> Test your agents through multi-turn conversations with a simulated user to capture issues before production and prevent regressions.
**Quick setup?** Paste the [Connect your agent prompt](/docs/agent-testing/connect-your-agent) into your coding agent to connect your agent's HTTP endpoint and run a first test suite, or the [scenarios prompt](/docs/skills/code-prompts#add-scenario-tests) to add code-first scenario tests.
## What agent testing is for
Agent testing ensures your agent handles real conversations correctly before they reach production. You describe a situation and the criteria the agent must meet, and LangWatch runs a simulated user through the conversation, then a judge gives a pass or fail for each criterion with its reasoning.
For example, a customer asks for a refund on an order from last year. The criteria: the agent must look up the order and must not promise the refund. The simulated user plays the customer with different words on every run, and the judge checks each criterion against the conversation and the agent's own traces.
Use it to:
* Check the agent before a release. Run the test suite and read the pass rate.
* Turn a production incident into a permanent test, so the same mistake does not come back.
* Catch a regression when a prompt, a model or a tool changes.
## How a test runs
| Part | What it does |
| -------------- | ---------------------------------------------------------------------------------------------------- |
| Simulated user | Plays the user in the situation the scenario describes, with different wording on every run. |
| Your agent | Answers each turn as it does in production. |
| Judge | Reads the conversation and the agent's traces, then passes or fails each criterion and explains why. |
A scenario is one situation and its criteria. A [test suite](/docs/agent-testing/test-suites) groups the scenarios you run together. A [run plan](/docs/agent-testing/run-plans) is the history of one configuration, and the [Results](/docs/agent-testing/results) tab shows its pass rate over time.
## Two ways to run
Both use the same simulated user, the same judge and the same results pages. The difference is where you write the scenarios.
Connect your agent over HTTP, then write, run and see scenarios in the LangWatch interface. No code needed.
Write scenarios with the Scenario SDK in Python or TypeScript, run them with your test runner and in CI, and see every run in LangWatch.
## What you can test
Multi-turn conversations judged on outcomes, internal actions and the things the agent must never do.
Real audio through ElevenLabs, OpenAI Realtime, Twilio, Pipecat and Gemini Live, with background noise, interruptions and latency metrics.
Multi-turn adversarial attacks with escalation, refusal detection and backtracking, to find security gaps before a release.
## Agent testing vs evaluations
An evaluation scores one input against one expected output, on a dataset. It fits one step inside the agent, such as a retrieval step or one prompt, where you optimize a metric on many examples.
For example, an evaluation dataset has rows like these:
| query | expected\_answer |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| What is your refund policy? | We offer a 30-day money-back guarantee on all purchases. |
| How do I cancel my subscription? | You can cancel your subscription by logging into your account and clicking the "Cancel Subscription" button. |
An agent test scores a whole conversation. The scenario describes the user's situation, and the criteria check each step, including the tools the agent called:
```python theme={null}
script=[
scenario.user("hey I have a problem with my order"),
scenario.agent(),
expect_ticket_created(),
expect_ticket_label("ecommerce"),
scenario.user("i want a refund!"),
scenario.agent(),
expect_tool_call("search_policy"),
scenario.user("this is ridiculous! let me talk to a human being"),
scenario.agent(),
expect_tool_call("escalate_to_human"),
]
```
Use evaluations for the parts inside the agent. Use agent testing to check the agent end to end: before a release, to reproduce an incident, and to catch a regression. Both run in the same LangWatch project.
[Scenario](https://langwatch.ai/scenario/) is the most advanced agent testing framework available. The platform and the SDKs both run on it: the same simulated user and the same judge test text, voice and adversarial conversations, in Python, TypeScript and Go, against any agent framework.
## Go deeper
The situation, the criteria, labels, parameters, versions and the editor.
Group scenarios, run a group, and where scenarios written in code appear.
What happens when you press Run, and how runs join one history.
The Results tab, the run detail, the conversation drawer and what each status means.
The same scenarios against two agents, or one agent on two settings, side by side.
Register the HTTP endpoint and let the judge read the agent's own traces.
Start a test suite from a CI job and fail the job on a failed run.
The full reference of the Scenario framework.