Skip to main content

​
The Results tab

Test Runs shows the number of scenario executions in the period. The period is Last 7 days, Last 15 days or Last 30 days; runs older than 30 days are in cold storage and do not appear here.
The Results tab grouped by run plan, with the pass rate and the trend of each plan
Group by sets what a row is: The filters narrow every grouping: Scenario, Label, Target and All statuses, Passed or Failed. Reset filters clears them. Pass is green at 100%, amber from 40%, red below, and grey with no result. Trend shows the last runs as bars, oldest first, at most 14. Charts opens the tiles Executions, Pass rate, Failing scenarios and Cost for the period, and Pass rate over time.
The Charts block with the four tiles and the pass rate over time

​
A run plan and its runs

Press a run plan. The sidebar lists its runs as Run #n with the note, the time and the pass rate; Load More… loads older ones. The header of the selected run shows the pass rate, the total duration and the total cost. Hover it for Completed, Avg Agent Latency, Avg Agent Cost, Total Duration and Total Cost, with percentiles behind the two averages.
A run plan with its runs in the sidebar, the summary pills and the table of results
Show run settings opens the configuration the run used: Started, Targets, Parameters, Repeat, Simulator model and Judge model. The Table view has one row per scenario execution: Result, Scenario, Evaluators and Time · cost. The Evaluators column appears only on a run that has evaluators, and its cell shows one pill per attached evaluator, with its verdict, its score or its status when it was skipped or hit an error; a row whose evaluators are still running reads Grading. The row menu offers Open the conversation, Rerun this scenario and Edit scenario. The Grid view shows the same executions as cards, one colour per result, each ending in Completed or Failed, or in Evaluating while the evaluators of the run are still reporting.
The Grid view of a run, one card per scenario execution
Export as CSV downloads the run. Run again starts a new run with the plan’s configuration, and Stop all cancels the conversations still running.

​
A conversation

Press a row to open the drawer. The header shows the Status, the criteria count, the Duration, the Cost and when it Ran. The transcript follows, with View trace on every turn whose trace has arrived.
The conversation drawer of a failed run, with the failed criteria and the judge reasoning
Under the transcript: FAILED CRITERIA, PASSED CRITERIA and JUDGE REASONING, then Evaluators when the suite or the plan attached any, then Parameters with the values the run used, a secret parameter masked. Edit Scenario opens the scenario in the editor.

​
Evaluator results

An evaluator attached to the test suite or to the run plan runs after the judge, and the run keeps one result per evaluator. The drawer lists them under the criteria, one row each: the status, the name, a mark when it is required, the verdict or the score, and the reason under it.
The results page of a run: the header pills SQL Query Equivalence 67%, Answer clarity score 9.25 and PII Leak Scanner 100%, and one pill per evaluator on each row, one of them Skipped and one Fail
The header shows one pill per evaluator, and the Evaluators column shows one pill per evaluator on every row. A skipped evaluator keeps its pill, in grey.
The run drawer of a failed run: the verdict line reads FAILED · SQL Query Equivalence, the passed criterion, and the evaluator rows with the failed required SQL Query Equivalence and its explanation, a 7.00 clarity score and a passed PII Leak Scanner
A run the required SQL Query Equivalence failed: the verdict line names it, its row shows the explanation and the Inputs, and the criterion the judge passed is kept above. The run header shows a pill per evaluator: the pass rate for one with a verdict, the mean for one with a score. The verdict of a run is settled when the evaluators are in. A run whose evaluators all pass, or that has none, keeps the judge’s verdict. Between the judge’s verdict and the evaluator results the run reads as PENDING_EVALUATION, evaluating in the interface: the Result column shows Evaluating with a spinner and the Evaluators column shows Grading. langwatch test-suite run ... --wait and the batch counts treat such a run as still going, so a batch is not reported as green while a required evaluator can still fail one of its runs. The run stays pending until its evaluators are recorded. A grading job that is lost is recorded as errored evaluators once its deadline passes, so a required evaluator that never ran fails the run instead of leaving it pending. The evaluators a run is graded with are fixed when the run is scheduled. Editing the suite or the run plan while a batch executes changes the runs scheduled after the edit, never the ones already queued. The results reach the CLI and the API as results.evaluations on langwatch simulation-run get <scenarioRunId> --format json and GET /api/simulation-runs/{scenarioRunId}, one entry per evaluator with status, required, passed, score, label and details. While the run is going, the drawer shows The conversation is running… and then The judge is reading the conversation…, and the verdict appears in place when it is in.

​
What a status means

The Result column of the table shows Passed, Failed, Running or Evaluating. The Status chip of the drawer is more precise:
The conversation drawer of a run whose target could not be reached, with the error
A criterion the judge could not decide is inconclusive. A criterion about an internal action goes inconclusive when the agent’s traces did not arrive; see Linking your traces. Also check: Run plans, Compare agents, Run from CI for the batch API behind the same runs.
Last modified on September 13, 2026