Limits
Ask for more rows than your plan allows and the run is refused with
instant_eval_row_cap_exceeded, with the cap and the plan in the refusal. When the rows in the window outnumber the limit, the run judges up to the limit and reports isCapped, so narrow the window or the filter when the answer has to cover every row.
If a run makes no progress for fifteen minutes, LangWatch fails it rather than leaving it hanging, and you can still read the verdicts it wrote.
How long does a run take
Judging 1,000 conversations takes about nine seconds, and 10,000 about a minute. The rows are read in pages and each page is judged in parallel while the next one is read, so the time grows with the rows and with the length of their text rather than with the number of questions.What a run costs
The rate is 0.0546 USD per million input tokens the judge reads, and output is free. You pay for each row’s text plus your questions, in one judge request per row: more for a long conversation than for a short one, and about the same for ten questions as for one. For example, the first run on the overview judged 100 conversations, read 107,000 tokens and cost 0.0058 USD, so 10,000 conversations of the same size come to about 0.58 USD.Reading the cost before you start
Tokens: line of the finished run for what it actually read.
run --estimate does the same and exits. A plain run over more than 1,000 rows prints the estimate before it starts.
The free budget
Without a paid plan, your organization can spend 1 USD on Instant Evals in total, across all of its projects. Runestimate to see what is left, as freeBudgetRemainingUsd. Once you pass the budget, a new run or a judged langwatch query is refused with instant_eval_free_budget_exhausted, carrying spentUsd and budgetUsd, until you upgrade the organization. If a run is under way when the budget runs out, it stops there and you keep the verdicts it wrote.
A paid plan has no free budget: it is billed monthly for what was judged, up to the row cap. A gateway budget you set on the project, the team or the organization still applies, and stops a run when it is reached.
Where the spend shows up
Every finished run and every judged query is one row on the same spend ledger the LangWatch AI Gateway writes, labelled Instant Evals on the spend page. A gateway budget on a project, a team or the organization counts Instant Evals against it the same way it counts gateway traffic, so one budget caps both. If you cancel a run, you pay for the rows it judged before it stopped, and a run that judged no rows is free.What is stored
LangWatch stores the trace, the question and the answer for each verdict, and none of the judged text; read the text again withsample. Verdicts are kept until you delete the project, so a question asked about last quarter still has its answer next year.