Asking a first question
Pricing a run before you ask
run --estimate does the same and exits; a plain run over a
thousand rows prints the price first, then starts.
Take the row count as exact, and on a selection of ordinary conversations trust
the price to within a few percent. Because fifty rows are still a sample, expect
to pay more than it quoted on a selection holding a handful of enormous rows
among small ones, and check the finished run for the tokens it actually read.
Reading what came back
sample re-reads a few rows with the text that was judged beside the verdict it
received, which is how you see what the judge saw. results reads a page of
judgements, narrowed by question, by whether they matched, and by whether the
judge answered at all.
run
Starts a run. Takes a question, or a statement of your own.
A modifier flag describes the question before it, so a line may ask two
questions that each carry their own criteria:
Several questions from a file
q1, q2 and so on. The answer goes into a column with that name, and every
judgement is filed under it.
What --filter accepts
--filter takes the same syntax as the trace explorer’s search bar. With a
--target, you can filter on the fields of the trace row itself.
Every other field sits outside the trace row, so a
--target refuses it by
name and lists what you can use instead.
Ask for any of those with
--sql, which reaches every table. To see where
each target puts the filter in its statement, read
the target shorthand.
Rerunning the statement a target wrote
run prints the statement it wrote. It binds its window as two parameters, so
running it again through langwatch query needs both:
--sql and two --param flags.
What you see while it runs
run prints the statement, then one progress line that updates in place:
status line that reads it back. run exits non-zero when the run did not
finish, so a script can tell a red run from a green one.
In -o json and -o agents the command still waits, and answers with one
document holding the run and the judgements of the rows it printed.
Starting a run without waiting
--detach creates the run, prints its id and returns. Read it back later with
status, or follow it with status <id> --wait.
status
list
--before and --before-id, which
take the created time and the id of the oldest run on the previous page.
results
Use the cursor to read the next page. It is the only way to page safely while
the run is still writing: you never see the same judgement twice, and you never
skip one.
sample
cancel
Output
Use the shared output flags on any subcommand:-o table|json|agents|yaml,
--json <fields> and --jq <path>. In a machine format the statement is a
field of the run rather than a printed block, and the progress line stays on
standard error so the document a script reads holds one object.