Skip to main content

​
Situation and criteria

The situation describes the user and what they want, in one or a few sentences. The simulated user plays it turn by turn and improvises the words, so a scenario is not a script. The criteria, one per line, are what the judge checks at the end of the conversation. Each one passes, fails or is inconclusive. For example, a checkout support agent gets this scenario:
Situation
Criteria
Write criteria about outcomes and actions, not wording. A criterion about an internal action, such as “looks up the order before answering”, is verified against the agent’s own traces, so the judge does not rely on what the reply claims. See Linking your traces.

​
The editor

New scenario on the Scenarios tab, or Edit on a row, opens the editor.
The scenario editor with the title, the test suite, the situation, the criteria and the labels
Save stores the scenario. Save & Run stores it and opens the run dialog. Open recent run opens the latest run of this scenario.

​
Customize scenario

Three chips under the criteria open optional blocks:
The Customize scenario chips expanded: parameters, min and max turns, and the model overrides
  • Add parameters declares run parameters as one line of name=default pairs, for example plan=free, locale=de. The situation, the criteria and the target read a value as {{ params.NAME }}. See Run parameters.
  • Define min and max turns bounds the conversation. Max turns defaults to 10. Min turns has no default: the judge can end the conversation as soon as it has a verdict.
  • Override models picks the User simulator and the Judge for this scenario alone. Without an override, the run’s models apply.

​
Versions

Every save is a new version. The editor shows the current one as v3 · History; open History to see every version with when it was saved and by whom. Restore on an older version writes its content forward as a new version, so every version stays in the history.
The version history of a scenario, with Restore on an older version
When someone else saves the scenario while you have it open, the editor tells you and offers to reload before you save over it.

​
Actions on a scenario

The row menu of a scenario offers Edit, Duplicate, Open recent runs, Move to suite… and Archive. Select several rows to move them together.
The row menu of a scenario
Archiving removes the scenario from the tables and from every future run. Its past runs stay on the Results tab.

​
Create scenarios with Langy

Langy, the assistant in the bottom corner of every page, writes and saves scenarios from what you paste: a support ticket, a complaint, acceptance criteria, or a transcript of a conversation that went wrong. Ask for the scenario and check it in the editor afterwards.

​
Create scenarios from your editor

Through the LangWatch MCP integration, a coding assistant such as Claude Code or Cursor reads and writes the scenarios of your project. The REST API does the same: POST /api/scenarios creates one, PATCH /api/scenarios/{id} updates one. See the Scenarios API reference.

​
Limits

Also check: Test suites, Run plans, Run parameters.
Last modified on August 30, 2026