> ## Documentation Index > Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details. # Run from CI > Start a test suite from a CI job and wait for every run of the batch to finish. ## How it works A CI job starts a test suite, stores the batch id from the response, then polls that batch until every run is finished. The CLI, the REST API and the SDKs all start the same run. ## From the CLI ```bash theme={null} langwatch test-suite run "Checkout regression" --target http:agent_abc123 --wait ``` `--wait` polls until the batch is complete and exits non-zero when a run failed, which is what fails the job. The test suite is named by its id or by its name. Add `--format json` (or `-o json`) to get one final document on stdout instead of the progress lines. It contains `outcome`, `tallies` and the per-run `results`, so a job step can read the verdict without parsing prose. ## From the REST API `POST /api/v1/test-suites/{id}/run` schedules one run for each scenario of the test suite, against each target you name: ```bash theme={null} curl -X POST "https://app.langwatch.ai/api/v1/test-suites/suite_abc123/run" \ -H "X-Auth-Token: ${LANGWATCH_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "targets": [{ "type": "http", "referenceId": "agent_abc123" }], "note": "checkout rewrite" }' ``` The body takes `targets` and, when you want them, `name`, `repeatCount`, `simulatorModel`, `judgeModel`, `parameters`, `note` and `idempotencyKey`. Without `name`, the run goes under the run plan named after the test suite and the target, so every run of the same test suite against the same agent joins one history. Repeat the same `idempotencyKey` to make a retried job join the first run instead of starting a second one. The response has the values the CI job needs: ```json theme={null} { "scheduled": true, "batchRunId": "batch_xyz789", "setId": "set_abc123", "jobCount": 2, "runPlanId": "plan_abc123", "planName": "Checkout regression Support Agent", "created": false, "platformUrl": "https://app.langwatch.ai/my-project/agent-testing/results/checkout-regression-support-agent", "skippedArchived": { "scenarios": [], "targets": [] }, "items": [ { "scenarioRunId": "run_1", "scenarioId": "scn_1", "target": { "type": "http", "referenceId": "agent_abc123" }, "name": "Cancel on the free plan" }, { "scenarioRunId": "run_2", "scenarioId": "scn_2", "target": { "type": "http", "referenceId": "agent_abc123" }, "name": "Cancel on the enterprise plan" } ] } ``` `batchRunId` identifies the batch. `jobCount` is the number of runs the batch gets. `created` says whether this run created the plan or joined one that already had the name. To run a configuration of your own instead of a whole test suite, `POST /api/v1/run-plans/run` takes a `config` object with the scope, the targets, the repeat count and the two models. The scope is one of `{ "mode": "all" }`, `{ "mode": "test_suites", "testSuiteIds": [...] }`, `{ "mode": "labels", "labels": [...] }`, or `{ "mode": "scenarios" }` with `scenarioIds` beside it. `GET /api/v1/run-plans` lists the plans, and `POST /api/v1/run-plans/{id}/run` runs one again with its stored configuration. The `/api/suites` family still answers and is deprecated. Move CI jobs to `/api/v1/test-suites` and `/api/v1/run-plans`. ## From Python ```python theme={null} import langwatch langwatch.setup() result = langwatch.test_suites.run( "suite_abc123", targets=[{"type": "http", "referenceId": "agent_abc123"}], note="checkout rewrite", ) print(result["batchRunId"]) ``` `langwatch.run_plans.run(...)` starts a plan of your own, with `scope`, `targets`, `repeat_count`, `simulator_model` and `judge_model`. `scope="test_suites"` takes `test_suite_ids=[...]`, and `scope="scenarios"` takes `scenario_ids=[...]`. ## From TypeScript ```typescript theme={null} import { LangWatch } from "langwatch"; const langwatch = new LangWatch(); const result = await langwatch.testSuites.run("suite_abc123", { targets: [{ type: "http", referenceId: "agent_abc123" }], note: "checkout rewrite", }); console.log(result.batchRunId); ``` `langwatch.runPlans.run({ ... })` takes the same body as `POST /api/v1/run-plans/run`. ## Compare two agents, or one agent on two settings A run goes against every target you name, so naming two targets runs each scenario against both and stores the results under one batch, one column per target on the results page. A target can also have its own parameter values, so the same agent named twice with different values is a comparison of that agent on two settings. [Compare agents](/docs/agent-testing/compare-agents) covers how the platform shows them. ```bash theme={null} langwatch test-suite run "Checkout regression" \ --target 'http:agent_abc123?model=gpt-5' \ --target 'http:agent_abc123?model=gpt-5-mini' \ --wait ``` The suffix after the question mark is a query string. Both halves are percent-decoded, so a value that contains `?` or `&` must be written as `%3F` or `%26`. A value is read as the type it looks like: `true` and `false` become booleans, a plain number becomes a number, everything else stays text. On the REST API the same values go in `runParameters` on the target: ```bash theme={null} curl -X POST "https://app.langwatch.ai/api/v1/test-suites/suite_abc123/run" \ -H "X-Auth-Token: ${LANGWATCH_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "targets": [ { "type": "http", "referenceId": "agent_abc123", "runParameters": { "model": "gpt-5" } }, { "type": "http", "referenceId": "agent_abc123", "runParameters": { "model": "gpt-5-mini" } } ], "note": "gpt-5 against gpt-5-mini" }' ``` `runParameters` are merged over the run-level `parameters`, and the target wins. Use `parameters` for what every target shares, such as a fixture id or a tenant, and `runParameters` for what differs between the targets. ## Poll the batch `GET /api/simulation-runs/batches/{batchRunId}` answers with the counts of one batch: ```bash theme={null} curl "https://app.langwatch.ai/api/simulation-runs/batches/batch_xyz789" \ -H "X-Auth-Token: ${LANGWATCH_API_KEY}" ``` ```json theme={null} { "batchRunId": "batch_xyz789", "totalCount": 2, "passCount": 1, "failCount": 0, "runningCount": 1, "settledCount": 1, "stalledCount": 0, "lastRunAt": 1755691200000, "lastUpdatedAt": 1755691260000, "firstCompletedAt": 1755691230000, "allCompletedAt": null, "isComplete": false } ``` | Field | Meaning | | -------------- | -------------------------------------------------------------------------------------------- | | `totalCount` | Runs in the batch now. | | `runningCount` | Runs that are queued or in progress. | | `settledCount` | Runs that are neither queued nor in progress: passed, failed, errored, cancelled or stalled. | | `isComplete` | True when `settledCount` equals `totalCount`. | Stop the poll when `isComplete` is true **and** `totalCount` is at least `jobCount`. The two conditions go together: the platform creates the runs asynchronously, so a batch read right after the trigger can have fewer runs than `jobCount`, and a batch of two created runs out of six reports itself complete. The same asynchronous creation makes the first read answer `404`. The batch becomes readable when its first run is stored, which is a moment after the trigger returns. Treat a `404` as "not stored yet" and poll again until your own timeout expires. A `404` is a missing batch only after that timeout. Every other status stops the job at once. A wrong token answers `401` and a failed read answers `5xx`, and a loop that polls through them reports a batch timeout thirty minutes later for a fault that was clear on the first read. Exit the CI job nonzero when `failCount` is above zero **or** `stalledCount` is above zero. A stalled run settles without passing, so it counts in `settledCount` but not in `failCount`, and a job that reads `failCount` alone reports success for a batch that never finished its work. Open the batch in the platform for the run details. ```bash theme={null} # Poll until the batch is complete, then set the exit code. deadline=$(( $(date +%s) + 1800 )) while [ "$(date +%s)" -lt "$deadline" ]; do body=$(curl -s -w '\n%{http_code}' --connect-timeout 10 --max-time 30 \ "https://app.langwatch.ai/api/simulation-runs/batches/${BATCH_RUN_ID}" \ -H "X-Auth-Token: ${LANGWATCH_API_KEY}") status=$(printf '%s' "$body" | tail -n1) summary=$(printf '%s' "$body" | sed '$d') case "$status" in 200) if [ "$(printf '%s' "$summary" | jq '.isComplete')" = "true" ] \ && [ "$(printf '%s' "$summary" | jq ".totalCount >= ${JOB_COUNT}")" = "true" ]; then printf '%s' "$summary" | jq -e '.failCount == 0 and .stalledCount == 0' exit $? fi ;; # 404 is a batch whose first run is not stored yet, 000 is a request that # did not complete. Both are read again. 404|000) ;; *) echo "The batch read answered ${status}: ${summary}" exit 1 ;; esac sleep 10 done echo "The batch did not finish before the timeout." exit 1 ``` The request has its own time limits, so a read that hangs cannot block the job. The deadline is read between requests, so the job stops within one request and one sleep of it, which is 40 seconds with the values above. `langwatch test-suite run ... --wait` does the same poll, and its exit code follows the results. ## Give the batch a note `--note` stores one short line with the batch: why it was run, or what changed. Every run in the batch has it, and the platform shows it beside the run. ```bash theme={null} langwatch test-suite run suite_abc123 --target http:agent_abc123 \ --note "$(git log -1 --pretty=%s | cut -c1-200)" --wait ``` The note is free text, up to 200 characters, and a longer one is refused before the batch is queued. The `cut` above trims a long commit subject to that limit. `langwatch simulation-run list` and `langwatch simulation-run get` show the note back.