> ## Documentation Index > Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details. # Instant Evals CLI > Every langwatch instant-eval subcommand and flag, with the lines that start a run, follow it, read its matches and sample the text that was judged. ## Asking a first question ```bash theme={null} langwatch instant-eval run "the customer sounds annoyed" \ --target threads --last 7d --limit 100 ``` That judges a hundred of the last seven days of conversations and prints the matches when it is done. While it works you see the LangWatchQL statement it ran, which is the statement you edit when you want more than the shorthand gives you, and a progress line. ## Pricing a run before you ask ```bash theme={null} langwatch instant-eval estimate "the customer sounds annoyed" \ --target threads --last 30d --limit 10000 ``` The estimate counts the rows, measures fifty of their texts spread across the whole selection, and prices the run from that. No row is judged and no charge is made. `run --estimate` does the same and exits; a plain `run` over a thousand rows prints the price first, then starts. Take the row count as exact, and on a selection of ordinary conversations trust the price to within a few percent. Because fifty rows are still a sample, expect to pay more than it quoted on a selection holding a handful of enormous rows among small ones, and check the finished run for the tokens it actually read. ## Reading what came back ```bash theme={null} langwatch instant-eval sample -n 5 langwatch instant-eval results --matched -o json ``` `sample` re-reads a few rows with the text that was judged beside the verdict it received, which is how you see what the judge saw. `results` reads a page of judgements, narrowed by question, by whether they matched, and by whether the judge answered at all. ## run Starts a run. Takes a question, or a statement of your own. | Flag | What it does | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `[question]` | The question, as the first argument | | `--ask ` | The same, as a flag. Repeat it to ask several questions of the same text | | `--criteria ` | For a yes or no question: what counts as yes, then what counts as no. Write it twice. Not combinable with `--threshold` on the same question | | `--score ` | Ask for a rating on that whole-numbered scale instead of a yes or no | | `--category ` | Ask which option fits. Repeat it for each option | | `--threshold ` | Where a yes or no question's probability becomes a pass. Not combinable with `--criteria` on the same question | | `--id ` | What to call the question before it, which becomes its column | | `--questions-file ` | Read the questions from a JSON or YAML file | | `--target ` | What one row is: `traces`, `threads` or `llm-spans` (default `traces`) | | `--filter ` | Narrow the rows with a trace filter | | `--last ` | How far back to look: `7d`, `24h`, `30m`, `2w` (default `7d`) | | `--start ` `--end ` | The window, exactly | | `--sql ` `--sql-file ` | Run a LangWatchQL statement of your own | | `--param ` | Bind one of your statement's parameters | | `--limit ` | Rows the run may judge (default 1000) | | `--name ` | What to call the run | | `--estimate` | Price the run and exit | | `--detach` | Create the run and return its id instead of waiting for it | | `--show ` | Rows to print when the run finishes (default 20, at most 25) | A modifier flag describes the question before it, so a line may ask two questions that each carry their own criteria: ```bash theme={null} langwatch instant-eval run \ --ask "the customer sounds annoyed" \ --criteria "sarcasm, repetition or a raised voice count" \ --criteria "a calm complaint does not count" \ --ask "the agent apologised" --threshold 0.8 \ --target threads --last 7d ``` ### Several questions from a file ```json theme={null} [ { "id": "annoyed", "kind": "boolean", "instructions": "the customer sounds annoyed" }, { "id": "intent", "kind": "category", "instructions": "what is being asked for", "options": [ { "name": "refund", "description": "wants money back" }, { "name": "bug", "description": "reports something broken" } ] } ] ``` ```bash theme={null} langwatch instant-eval run --questions-file questions.json --target threads --last 7d ``` A file is also how you name a question. Without a name, questions are called `q1`, `q2` and so on. The answer goes into a column with that name, and every judgement is filed under it. ### What `--filter` accepts `--filter` takes the same syntax as the trace explorer's search bar. With a `--target`, you can filter on the fields of the trace row itself. | Group | Fields | | -------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Identity | `traceId`, `traceName`, `service`, `origin`, `user`, `customer`, `conversation`, `scenarioRun` | | Classification | `topic`, `subtopic`, `label`, `model`, `status:error` | | Cost and size | `cost`, `duration`, `tokens`, `promptTokens`, `completionTokens`, `tokensPerSecond`, `ttft`, `ttlt`, `spans`, `tokensEstimated` | | Prompts | `selectedPrompt`, `lastUsedPrompt`, `promptVersion` | | Attributes | `trace.attribute.` | | Free text | a bare word | Every other field sits outside the trace row, so a `--target` refuses it by name and lists what you can use instead. | Group | Refused fields | | ---------------------- | ---------------------------------------------------------------------------------------------------- | | Spans | `spanId`, `spanName`, `spanType`, `spanStatus`, `rootSpanType`, `span.attribute.`, `containsAi` | | Evaluations | `eval`, `evaluator*`, `guardrail` | | Events and annotations | `event`, `event.attribute.`, `annotation`, `feedback` | | Scenario runs | `scenario`, `scenarioSet`, `scenarioBatch`, `scenarioStatus`, `scenarioVerdict` | | Errors | `status:ok`, `status:warning` | Ask for any of those with `--sql`, which reaches every table. To see where each target puts the filter in its statement, read [the target shorthand](/docs/features/instant-evals/shorthand). ### Rerunning the statement a target wrote `run` prints the statement it wrote. It binds its window as two parameters, so running it again through `langwatch query` needs both: ```bash theme={null} langwatch query "$(cat written.sql)" \ --param start_at=2026-09-11T00:00:00 \ --param end_at=2026-09-18T00:00:00 ``` Change the values to move the window without touching the query. Resubmitting it as a run works the same way, with `--sql` and two `--param` flags. ### What you see while it runs `run` prints the statement, then one progress line that updates in place: ``` Judging 3,200/10,000 · 412 matched · 1.9M tokens · 12s ``` When the run ends, that is replaced by the headline and the first rows: ``` Found 412 matches in 10,000 conversations · 20.4s · 6.1M tokens · $0.33 ``` Press Ctrl-C to stop watching. The run keeps going, and the command prints the `status` line that reads it back. `run` exits non-zero when the run did not finish, so a script can tell a red run from a green one. In `-o json` and `-o agents` the command still waits, and answers with one document holding the run and the judgements of the rows it printed. ### Starting a run without waiting ```bash theme={null} langwatch instant-eval run "the customer sounds annoyed" --target threads --detach ``` `--detach` creates the run, prints its id and returns. Read it back later with `status`, or follow it with `status --wait`. ## status ```bash theme={null} langwatch instant-eval status --wait ``` Reads one run: where it is, how many rows it found and judged, how many matched in total and per question, what the judge could not answer, and the tokens and price the judging came to. Prints the statement too. ## list ```bash theme={null} langwatch instant-eval list -o json ``` The project's runs, newest first. Page with `--before` and `--before-id`, which take the created time and the id of the oldest run on the previous page. ## results ```bash theme={null} langwatch instant-eval results --question annoyed --matched --limit 200 ``` | Flag | What it does | | --------------------------- | --------------------------------------------------------- | | `--question ` | Only that question's judgements | | `--matched` / `--unmatched` | Only the ones that matched, or only the ones that did not | | `--status ` | Only `judged`, `skipped` or `failed` judgements | | `--limit ` | Judgements per page, at most 1000 (default 100) | | `--cursor ` | The cursor you got from the previous page | Use the cursor to read the next page. It is the only way to page safely while the run is still writing: you never see the same judgement twice, and you never skip one. ## sample ```bash theme={null} langwatch instant-eval sample -n 5 ``` A few of the run's rows with the judged text beside the verdict. The text is re-read through the statement's own extraction functions, so sampling judges no row again and costs no money. ## cancel ```bash theme={null} langwatch instant-eval cancel ``` Stops the run before its next page, keeping every judgement it had already written for you to read the same way. Cancel a run that has already finished and you get back a 409. ## Output Use the shared output flags on any subcommand: `-o table|json|agents|yaml`, `--json ` and `--jq `. In a machine format the statement is a field of the run rather than a printed block, and the progress line stays on standard error so the document a script reads holds one object.