Langy writes and runs Scenario simulation tests and evaluations through the langwatch CLI, so they live in your project like any test or evaluator you wrote by hand. Which one it picks depends on what the question needs to prove; see the table below.
Also check: Ask Langy (how you request a test), Pull requests (the tests it wrote run in your CI), Agent Testing and Evaluations (the product surfaces these land in).
Scenario simulation tests
A Scenario test is a multi-turn simulation. A simulated user drives your agent across a full conversation, probing it the way a real user would, and a judge scores the transcript against the criteria you set.
This is how Langy catches behavior that a single-turn check misses: an agent that holds the line for one message but drifts by turn 15.
Langy writes and runs these with:
langwatch scenario createlangwatch scenario runThe runs land in Agent Testing > Results, where you can filter, group, and open any run.

Open a run to see the judge's verdict and its reasoning against your criteria.

When a run fails, the detail includes the bug log so you can see exactly where the agent broke.

Read more in the Agent Testing docs: Overview, Write scenarios in code, Red teaming, and Voice agents.
Evaluations
An evaluation scores a run against a criterion: answer correctness, a custom rule, an LLM judge, a pairwise comparison. Langy writes evaluators and runs experiments and monitors with the CLI:
langwatch evaluator createlangwatch experiment runlangwatch monitor createUse each for a different job:
langwatch experiment runscores an offline or batch run, for comparing two models or two prompts. See Experiments and running evals in CI/CD.langwatch monitor createsets up an online evaluation that scores live production traffic. See Online evaluation.langwatch evaluator createdefines the evaluator both of the above use. See Evaluators.
Which one Langy picks
Langy picks based on what your question needs to prove:
| You want to check | Langy writes |
|---|---|
| The agent handles a full conversation correctly | A Scenario simulation test |
| One run scores well on a criterion | An evaluation |
| Live traffic stays within a criterion | An online monitor |
| Two models or prompts, head to head | An experiment |
You can also name the one you want in the chat: "write a Scenario test that..." or "set up an evaluator for...".