import requests
url = "https://app.langwatch.ai/api/v1/instant-evals"
payload = { "name": "<string>" }
headers = {
"X-Auth-Token": "<api-key>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)const options = {
method: 'POST',
headers: {'X-Auth-Token': '<api-key>', 'Content-Type': 'application/json'},
body: JSON.stringify({name: '<string>'})
};
fetch('https://app.langwatch.ai/api/v1/instant-evals', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));curl --request POST \
--url https://app.langwatch.ai/api/v1/instant-evals \
--header 'Content-Type: application/json' \
--header 'X-Auth-Token: <api-key>' \
--data '
{
"name": "<string>"
}
'{
"id": "<string>",
"name": "<string>",
"sql": "<string>",
"parameters": {},
"questions": [
{
"id": "<string>",
"function": "<string>",
"kind": "boolean",
"reads": "<string>",
"threshold": 123
}
],
"limit": 123,
"status": "queued",
"total": 123,
"progress": 123,
"matched": 123,
"matchedByQuestion": {},
"failed": 123,
"skipped": 123,
"tokens": 123,
"priceUsd": 123,
"error": "<string>",
"createdAt": "<string>",
"updatedAt": "<string>",
"startedAt": "<string>",
"finishedAt": "<string>"
}Start a run
Start a run. The statement is accepted, its questions are derived from the eval functions it projects, and the judging happens on the queue: the answer is the queued run, and its progress is read back from the run endpoint. A statement the query policy refuses, one that projects no TraceId, one that projects no eval function, and a row limit past what the plan allows are all refused before anything is judged. Instead of a statement you may send a target and your questions, and the statement is written for you and handed back on the run; sending both is refused.
import requests
url = "https://app.langwatch.ai/api/v1/instant-evals"
payload = { "name": "<string>" }
headers = {
"X-Auth-Token": "<api-key>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)const options = {
method: 'POST',
headers: {'X-Auth-Token': '<api-key>', 'Content-Type': 'application/json'},
body: JSON.stringify({name: '<string>'})
};
fetch('https://app.langwatch.ai/api/v1/instant-evals', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));curl --request POST \
--url https://app.langwatch.ai/api/v1/instant-evals \
--header 'Content-Type: application/json' \
--header 'X-Auth-Token: <api-key>' \
--data '
{
"name": "<string>"
}
'{
"id": "<string>",
"name": "<string>",
"sql": "<string>",
"parameters": {},
"questions": [
{
"id": "<string>",
"function": "<string>",
"kind": "boolean",
"reads": "<string>",
"threshold": 123
}
],
"limit": 123,
"status": "queued",
"total": 123,
"progress": 123,
"matched": 123,
"matchedByQuestion": {},
"failed": 123,
"skipped": 123,
"tokens": 123,
"priceUsd": 123,
"error": "<string>",
"createdAt": "<string>",
"updatedAt": "<string>",
"startedAt": "<string>",
"finishedAt": "<string>"
}Authorizations
Project API key for sending traces and accessing project-scoped resources. Format: sk-lw-... (no underscore). Obtain one by creating a project via the Admin API or the LangWatch UI.
Body
The LangWatchQL statement to judge. It must project TraceId and at least one eval function column. Send this or target, never both.
1 - 50000Values for the parameters the statement declares.
Show child attributes
Show child attributes
What one judged row is, in place of a statement: a trace, a conversation, or one model call. The statement is written for you from this and the questions, and handed back on the run so you can edit it and resubmit.
traces, threads, llm_spans With target: a trace filter, in the language the trace explorer's search bar speaks, narrowing which rows are judged.
4000With target: the oldest instant to judge, as an ISO 8601 timestamp. Defaults to seven days ago.
With target: the newest instant to judge. Defaults to now.
With target: what to ask of each row. One classification asks them all, which is why a three-question run costs about what a one-question run does.
Show child attributes
Show child attributes
What to call the run. Yours to choose.
1 - 200Rows the run may judge. Ten thousand by default on every plan, up to one hundred thousand on a plan that lifts the cap.
x <= 100000Response
Success
The run id.
What the run was called, if anything.
The statement, exactly as submitted.
The values the statement's parameters were filled with.
Show child attributes
Show child attributes
One entry per eval function the statement projects, derived from it when the run was accepted.
Show child attributes
Show child attributes
Rows this run may judge.
Where the run is in its life.
queued, planning, running, finished, failed, cancelled Rows the run found, bounded by its limit. Null until it has looked.
Rows judged so far.
Judgements that matched, across this run's boolean questions. Null when the run asked none: a score or a category question has no match to count.
Per question: matches for a boolean question, judged rows for a score or a category one.
Show child attributes
Show child attributes
Rows the judge could not answer.
Rows the judge declined to answer.
Input tokens the judge billed for.
What the judging costs you, in United States dollars.
The code of the failure that ended the run, when one did.
When the run was accepted.
When the run was last written to.
When the run began reading rows.
When the run ended.
Was this page helpful?