Run from CI

How it works

A CI job starts a test suite, stores the batch id from the response, then polls that batch until every run is finished. The CLI, the REST API and the SDKs all start the same run.

From the CLI

langwatch test-suite run "Checkout regression" --target http:agent_abc123 --wait

--wait polls until the batch is complete and exits non-zero when a run failed, which is what fails the job. The test suite is named by its id or by its name. Add --format json (or -o json) to get one final document on stdout instead of the progress lines. It contains outcome, tallies and the per-run results, so a job step can read the verdict without parsing prose.

From the REST API

POST /api/v1/test-suites/{id}/run schedules one run for each scenario of the test suite, against each target you name:

curl -X POST "https://app.langwatch.ai/api/v1/test-suites/suite_abc123/run" \
  -H "X-Auth-Token: ${LANGWATCH_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "targets": [{ "type": "http", "referenceId": "agent_abc123" }],
    "note": "checkout rewrite"
  }'

The body takes targets and, when you want them, name, repeatCount, simulatorModel, judgeModel, parameters, note and idempotencyKey. Without name, the run goes under the run plan named after the test suite and the target, so every run of the same test suite against the same agent joins one history. Repeat the same idempotencyKey to make a retried job join the first run instead of starting a second one. The response has the values the CI job needs:

{
  "scheduled": true,
  "batchRunId": "batch_xyz789",
  "setId": "set_abc123",
  "jobCount": 2,
  "runPlanId": "plan_abc123",
  "planName": "Checkout regression Support Agent",
  "created": false,
  "platformUrl": "https://app.langwatch.ai/my-project/agent-testing/results/checkout-regression-support-agent",
  "skippedArchived": { "scenarios": [], "targets": [] },
  "items": [
    {
      "scenarioRunId": "run_1",
      "scenarioId": "scn_1",
      "target": { "type": "http", "referenceId": "agent_abc123" },
      "name": "Cancel on the free plan"
    },
    {
      "scenarioRunId": "run_2",
      "scenarioId": "scn_2",
      "target": { "type": "http", "referenceId": "agent_abc123" },
      "name": "Cancel on the enterprise plan"
    }
  ]
}

batchRunId identifies the batch. jobCount is the number of runs the batch gets. created says whether this run created the plan or joined one that already had the name. To run a configuration of your own instead of a whole test suite, POST /api/v1/run-plans/run takes a config object with the scope, the targets, the repeat count and the two models. The scope is one of { "mode": "all" }, { "mode": "test_suites", "testSuiteIds": [...] }, { "mode": "labels", "labels": [...] }, or { "mode": "scenarios" } with scenarioIds beside it. GET /api/v1/run-plans lists the plans, and POST /api/v1/run-plans/{id}/run runs one again with its stored configuration.

The /api/suites family still answers and is deprecated. Move CI jobs to /api/v1/test-suites and /api/v1/run-plans.

From Python

import langwatch

langwatch.setup()

result = langwatch.test_suites.run(
    "suite_abc123",
    targets=[{"type": "http", "referenceId": "agent_abc123"}],
    note="checkout rewrite",
)
print(result["batchRunId"])

langwatch.run_plans.run(...) starts a plan of your own, with scope, targets, repeat_count, simulator_model and judge_model. scope="test_suites" takes test_suite_ids=[...], and scope="scenarios" takes scenario_ids=[...].

From TypeScript

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();

const result = await langwatch.testSuites.run("suite_abc123", {
  targets: [{ type: "http", referenceId: "agent_abc123" }],
  note: "checkout rewrite",
});
console.log(result.batchRunId);

langwatch.runPlans.run({ ... }) takes the same body as POST /api/v1/run-plans/run.

Compare two agents, or one agent on two settings

A run goes against every target you name, so naming two targets runs each scenario against both and stores the results under one batch, one column per target on the results page. A target can also have its own parameter values, so the same agent named twice with different values is a comparison of that agent on two settings. Compare agents covers how the platform shows them.

langwatch test-suite run "Checkout regression" \
  --target 'http:agent_abc123?model=gpt-5' \
  --target 'http:agent_abc123?model=gpt-5-mini' \
  --wait

The suffix after the question mark is a query string. Both halves are percent-decoded, so a value that contains ? or & must be written as %3F or %26. A value is read as the type it looks like: true and false become booleans, a plain number becomes a number, everything else stays text. On the REST API the same values go in runParameters on the target:

curl -X POST "https://app.langwatch.ai/api/v1/test-suites/suite_abc123/run" \
  -H "X-Auth-Token: ${LANGWATCH_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "targets": [
      {
        "type": "http",
        "referenceId": "agent_abc123",
        "runParameters": { "model": "gpt-5" }
      },
      {
        "type": "http",
        "referenceId": "agent_abc123",
        "runParameters": { "model": "gpt-5-mini" }
      }
    ],
    "note": "gpt-5 against gpt-5-mini"
  }'

runParameters are merged over the run-level parameters, and the target wins. Use parameters for what every target shares, such as a fixture id or a tenant, and runParameters for what differs between the targets.

Poll the batch

GET /api/simulation-runs/batches/{batchRunId} answers with the counts of one batch:

curl "https://app.langwatch.ai/api/simulation-runs/batches/batch_xyz789" \
  -H "X-Auth-Token: ${LANGWATCH_API_KEY}"

{
  "batchRunId": "batch_xyz789",
  "totalCount": 2,
  "passCount": 1,
  "failCount": 0,
  "runningCount": 1,
  "settledCount": 1,
  "stalledCount": 0,
  "lastRunAt": 1755691200000,
  "lastUpdatedAt": 1755691260000,
  "firstCompletedAt": 1755691230000,
  "allCompletedAt": null,
  "isComplete": false
}

Field	Meaning
`totalCount`	Runs in the batch now.
`runningCount`	Runs that are queued or in progress.
`settledCount`	Runs that are neither queued nor in progress: passed, failed, errored, cancelled or stalled.
`isComplete`	True when `settledCount` equals `totalCount`.

Stop the poll when isComplete is true and totalCount is at least jobCount. The two conditions go together: the platform creates the runs asynchronously, so a batch read right after the trigger can have fewer runs than jobCount, and a batch of two created runs out of six reports itself complete. The same asynchronous creation makes the first read answer 404. The batch becomes readable when its first run is stored, which is a moment after the trigger returns. Treat a 404 as “not stored yet” and poll again until your own timeout expires. A 404 is a missing batch only after that timeout. Every other status stops the job at once. A wrong token answers 401 and a failed read answers 5xx, and a loop that polls through them reports a batch timeout thirty minutes later for a fault that was clear on the first read. Exit the CI job nonzero when failCount is above zero or stalledCount is above zero. A stalled run settles without passing, so it counts in settledCount but not in failCount, and a job that reads failCount alone reports success for a batch that never finished its work. Open the batch in the platform for the run details.

# Poll until the batch is complete, then set the exit code.
deadline=$(( $(date +%s) + 1800 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  body=$(curl -s -w '\n%{http_code}' --connect-timeout 10 --max-time 30 \
    "https://app.langwatch.ai/api/simulation-runs/batches/${BATCH_RUN_ID}" \
    -H "X-Auth-Token: ${LANGWATCH_API_KEY}")
  status=$(printf '%s' "$body" | tail -n1)
  summary=$(printf '%s' "$body" | sed '$d')

  case "$status" in
    200)
      if [ "$(printf '%s' "$summary" | jq '.isComplete')" = "true" ] \
        && [ "$(printf '%s' "$summary" | jq ".totalCount >= ${JOB_COUNT}")" = "true" ]; then
        printf '%s' "$summary" | jq -e '.failCount == 0 and .stalledCount == 0'
        exit $?
      fi
      ;;
    # 404 is a batch whose first run is not stored yet, 000 is a request that
    # did not complete. Both are read again.
    404|000) ;;
    *)
      echo "The batch read answered ${status}: ${summary}"
      exit 1
      ;;
  esac
  sleep 10
done

echo "The batch did not finish before the timeout."
exit 1

The request has its own time limits, so a read that hangs cannot block the job. The deadline is read between requests, so the job stops within one request and one sleep of it, which is 40 seconds with the values above. langwatch test-suite run ... --wait does the same poll, and its exit code follows the results.

Give the batch a note

--note stores one short line with the batch: why it was run, or what changed. Every run in the batch has it, and the platform shows it beside the run.

langwatch test-suite run suite_abc123 --target http:agent_abc123 \
  --note "$(git log -1 --pretty=%s | cut -c1-200)" --wait

The note is free text, up to 200 characters, and a longer one is refused before the batch is queued. The cut above trims a long commit subject to that limit. langwatch simulation-run list and langwatch simulation-run get show the note back.

Get Started

Agent Testing

Observability

Evaluations

Prompt Management

Platform

Examples & Cookbooks

Help

How it works

From the CLI

From the REST API

From Python

From TypeScript

Compare two agents, or one agent on two settings

Poll the batch

Give the batch a note

​How it works

​From the CLI

​From the REST API

​From Python

​From TypeScript

​Compare two agents, or one agent on two settings

​Poll the batch

​Give the batch a note

How it works

From the CLI

From the REST API

From Python

From TypeScript

Compare two agents, or one agent on two settings

Poll the batch

Give the batch a note