> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Running An Instant Eval

> Take a LangWatchQL statement that judges rows and run it as a job across your history, with progress, judgements you can page through, and a price you can see before you start.

## What A Run Is

An Instant Eval run takes one LangWatchQL statement and judges every row it matches. The statement is the same kind you send to `POST /api/v1/query`, with one or more eval function columns in its `SELECT` list. The query endpoint answers a page at a time; a run keeps going until it reaches the end of the statement or its row limit.

The statement is stored as submitted and handed back on every read, so an agent can fetch a run, change one predicate and submit the next one.

## The Statement A Run Accepts

Two things are required beyond what the query family already validates:

* The statement must project `TraceId`, because that is how a judgement is addressed.
* It must project at least one eval function column, such as `eval_passed(...) AS refused`.

Anything else the query validator refuses is refused here too, with the code `instant_eval_query_invalid` and the same violations in `meta`. A statement that validates but projects neither of the two required columns answers `instant_eval_query_missing_columns`, naming what is absent.

Project `SpanId` as well when your statement has one row per span rather than one per trace. That is what tells the run to address a judgement by the trace and the span together, so the spans of one trace come back as separate verdicts.

Add an `ORDER BY` when your statement carries a `LIMIT`. The run reads your statement once to count it and twice per page, and a `LIMIT` with no order lets the database return a different set of rows each time, so the pages are not guaranteed to cover one fixed set.

```sql theme={null}
SELECT
  TraceId,
  eval_passed(conversation(ConversationId), 'Did the agent refuse the request?') AS refused
FROM analytics.trace_metrics
WHERE OccurredAt >= subtractDays(now(), 30) AND ConversationId != ''
GROUP BY TraceId, ConversationId
```

## Pricing A Run Before You Start It

`POST /api/v1/instant-evals/estimate` takes the same body as a run and answers without judging anything. It counts the rows the statement matches, reads a sample of them to measure how much text a row carries, and multiplies:

```json theme={null}
{
  "rows": 8421,
  "rowsCapped": false,
  "avgTokens": 612,
  "totalTokens": 5153652,
  "requests": 8421,
  "costUsd": 0.216,
  "priceUsd": 0.281
}
```

`priceUsd` is what the run costs you and `costUsd` is what it costs us. One classification is made per row, carrying every question the statement asks, so a statement with four eval columns still sends one request per row.

An organization without a paid plan also gets `freeBudgetRemainingUsd`, what is left of its \$1 free budget. Once that is spent, a new run and a judged query answer `402` with `instant_eval_free_budget_exhausted`, carrying `spentUsd` and `budgetUsd` in `meta`, until the organization upgrades. The rate is on [Pricing](/docs/pricing#instant-evals).

## Row Limits

A run judges at most 10,000 rows by default. Pass `limit` to ask for fewer, or for up to 100,000 on a plan that lifts the cap. A `limit` above what your plan allows answers `instant_eval_row_cap_exceeded` with the cap and the plan in `meta`.

When the statement matches more rows than the limit, the run judges the first page-ordered rows up to the limit and reports `isCapped` as true. Narrow the statement rather than raising the limit when the answer has to cover everything.

## Starting A Run And Watching It

`POST /api/v1/instant-evals` accepts the statement and answers `202` with the run, whose `status` is `QUEUED`. Poll `GET /api/v1/instant-evals/{id}` for progress:

| Field               | What it says                                                                        |
| ------------------- | ----------------------------------------------------------------------------------- |
| `status`            | `QUEUED`, `PLANNING`, `RUNNING`, `FINISHED`, `FAILED` or `CANCELLED`                |
| `total`             | Rows the statement matched, once the run has counted them                           |
| `progress`          | Rows judged so far                                                                  |
| `matched`           | Judgements that matched, across the boolean questions. Null when the run asked none |
| `matchedByQuestion` | Matches for a boolean question, judged rows for a score or a category one           |
| `failed`            | Rows the judge could not answer                                                     |
| `skipped`           | Rows the judge declined, for instance because the text was too large                |
| `tokens`            | Input tokens billed so far                                                          |
| `priceUsd`          | What the run has cost you, written when it finishes                                 |

`GET /api/v1/instant-evals` lists the project's runs newest first, paged with `before` as an ISO 8601 cursor.

## Reading The Judgements

`GET /api/v1/instant-evals/{id}/results` pages through what the run wrote, keyset-paged with a `cursor`. Filter with `questionId`, `matched` and `status`:

```
GET /api/v1/instant-evals/run_123/results?questionId=refused&matched=true&limit=200
```

A judgement names the trace, the span, the question, the verdict and when it was made. Boolean questions carry `passed` and `probability`, score questions carry `score`, category questions carry `label` and the distribution in `probabilities`. A row the judge declined carries a `skippedReason` instead.

The judged text is not stored. `GET /api/v1/instant-evals/{id}/sample?n=5` re-reads a few rows through the original statement and returns them next to their verdicts, which is how you check a run answered the question you meant to ask. The rows differ between calls, and a run with a boolean question shows the rows that matched first.

## Cancelling

`POST /api/v1/instant-evals/{id}/cancel` stops a run that is still going. The page in flight finishes its current chunk, then the run ends as `CANCELLED` and is billed for what it judged. Cancelling a run that already ended answers `instant_eval_already_finished`.

## Querying Judgements Alongside Your Traces

Every judgement is also a row in the `judgments` analytics dataset, so a later query can join verdicts back to the traces that produced them:

```sql theme={null}
SELECT j.Label AS verdict, count() AS traces
FROM analytics.judgments AS j
WHERE j.RunId = {run:String} AND j.CreatedAt >= subtractDays(now(), 7)
GROUP BY verdict
ORDER BY traces DESC
```

`GET /api/v1/query/schema` lists the dataset's columns. Judgements are kept until you delete the project, because a retention window measured in weeks would remove the answer to a question asked about last quarter.

## Permissions

Reading runs and their results needs `analytics:view`. Starting or cancelling one needs `analytics:manage`.