What agent testing is for
Agent testing ensures your agent handles real conversations correctly before they reach production. You describe a situation and the criteria the agent must meet, and LangWatch runs a simulated user through the conversation, then a judge gives a pass or fail for each criterion with its reasoning. For example, a customer asks for a refund on an order from last year. The criteria: the agent must look up the order and must not promise the refund. The simulated user plays the customer with different words on every run, and the judge checks each criterion against the conversation and the agent’s own traces. Use it to:- Check the agent before a release. Run the test suite and read the pass rate.
- Turn a production incident into a permanent test, so the same mistake does not come back.
- Catch a regression when a prompt, a model or a tool changes.
How a test runs
A scenario is one situation and its criteria. A test suite groups the scenarios you run together. A run plan is the history of one configuration, and the Results tab shows its pass rate over time.
Two ways to run
Both use the same simulated user, the same judge and the same results pages. The difference is where you write the scenarios.Test from the platform
Connect your agent over HTTP, then write, run and see scenarios in the LangWatch interface. No code needed.
Write scenarios in code
Write scenarios with the Scenario SDK in Python or TypeScript, run them with your test runner and in CI, and see every run in LangWatch.
What you can test
Conversational agents
Multi-turn conversations judged on outcomes, internal actions and the things the agent must never do.
Voice agents
Real audio through ElevenLabs, OpenAI Realtime, Twilio, Pipecat and Gemini Live, with background noise, interruptions and latency metrics.
Red teaming
Multi-turn adversarial attacks with escalation, refusal detection and backtracking, to find security gaps before a release.
Agent testing vs evaluations
An evaluation scores one input against one expected output, on a dataset. It fits one step inside the agent, such as a retrieval step or one prompt, where you optimize a metric on many examples. For example, an evaluation dataset has rows like these:
An agent test scores a whole conversation. The scenario describes the user’s situation, and the criteria check each step, including the tools the agent called:
Go deeper
Scenarios
The situation, the criteria, labels, parameters, versions and the editor.
Test suites
Group scenarios, run a group, and where scenarios written in code appear.
Run plans
What happens when you press Run, and how runs join one history.
Results
The Results tab, the run detail, the conversation drawer and what each status means.
Compare agents
The same scenarios against two agents, or one agent on two settings, side by side.
Connect your agent
Register the HTTP endpoint and let the judge read the agent’s own traces.
Run from CI
Start a test suite from a CI job and fail the job on a failed run.
Scenario documentation
The full reference of the Scenario framework.