Skip to main content
Both are real artifacts, not drafts: Langy writes and runs them through the langwatch CLI, so what comes back lives in your project like any test or evaluator you wrote by hand. Which one it picks depends on what the question needs to prove; see the table below.
Also check: Ask Langy (how you request a test), Pull requests (the tests it wrote run in your CI), Agent Simulations and Evaluations (the product surfaces these land in).

​
Scenario simulation tests

A Scenario test is a multi-turn simulation. A simulated user drives your agent across a full conversation, probing it the way a real user would, and a judge scores the transcript against the criteria you set. This is how Langy catches behavior that a single-turn check misses: an agent that holds the line for one message but drifts by turn 15. Langy writes and runs these with:
The runs land in the Simulations surface, where you can filter, group, and open any run.
Simulations overview filtered to red-team runs

The Simulations overview filtered to red-team runs. Each row is a scenario run with its outcome.

Open a run to see the judge’s verdict and its reasoning against your criteria.
Scenario verdict and judge reasoning

A scenario run detail: the judge verdict with the reasoning it used to reach it.

When a run fails, the detail includes the bug log so you can see exactly where the agent broke.
Failed run detail with bug log

A failed-run detail with the bug log showing where the agent's behavior diverged from the criteria.

Read more in the Agent Simulations docs: Introduction, Getting started, Red teaming, and Voice agents.

​
Evaluations

An evaluation scores a run against a criterion: answer correctness, a custom rule, an LLM judge, a pairwise comparison. Langy writes evaluators and runs experiments and monitors with the CLI:
Use each for a different job:
  • langwatch experiment run scores an offline or batch run, for comparing two models or two prompts. See Experiments and running evals in CI/CD.
  • langwatch monitor create sets up an online evaluation that scores live production traffic. See Online evaluation.
  • langwatch evaluator create defines the evaluator both of the above use. See Evaluators.

​
Which one Langy picks

Langy picks based on what your question needs to prove: You can also name the one you want in the chat: “write a Scenario test that…” or “set up an evaluator for…”.