When to attach an evaluator
The judge grades behavior from the criteria. An evaluator compares against a known answer or runs a check an existing evaluator already does well: a SQL query against the golden one, a reply against a reference answer, a transcript through a PII scanner, retrieved contexts through a faithfulness check. Any evaluator of the project’s library can be attached, LLM judges included. A test suite declares fields for the values an evaluator compares against. A scenario may carry a value for each field; a blank value skips the evaluators that read it.Attach an evaluator
Edit suite on a test suite opens the suite editor. Under Customize test suite, Add evaluators shows the attached evaluators as chips and Add evaluator opens the evaluator list of the project. Picking one attaches it with its mappings inferred; when a required input is still unmapped, or when the evaluator reads an expected value, the evaluator drawer opens on the attachment so you confirm where each input reads from.Mappings
Every evaluator input reads from one source:
Mappings are inferred when an evaluator is attached. An input named
input, question or user_input reads the first user message; output, response or answer reads the last agent message; transcript, conversation or messages reads the transcript; contexts or retrieved_contexts reads the trace’s contexts.
An input named expected_output, expected_contexts or another expected_* reads a suite field. A field identifier equal to the input name wins. Otherwise the words of the identifier decide: expected_output takes a field carrying golden, answer, sql, query, label or target, which is how it takes golden_sql; expected_contexts takes one carrying schema, context or table, which is how it takes table_schema.
Two fields matching leaves the input unmapped, so you pick one in the drawer. With no match at all, a suite with a single field maps the input to it. A tool call or the spans are never inferred: set them by hand. spans hands a code or LLM evaluator the whole run, every span of every trace as JSON, when one tool call at a time is not enough.
The example of a text-to-SQL agent: the suite declares golden_sql and table_schema, and SQL Query Equivalence reads output from tool_calls.run_sql.input, expected_output from fields.golden_sql and expected_contexts from fields.table_schema.
Required evaluators and scores
Required to pass on the attachment makes a failing result fail the scenario. It is on by default for an evaluator that produces a pass or fail verdict. A score-only evaluator reports its score beside the verdict and never gates, whatever the switch says.What each run reports
After the conversation and the judge, the platform runs every attached evaluator and stores one result per evaluator on the run:
The run drawer lists the evaluators under the criteria, and the run header shows a pill per evaluator with its pass rate or its mean score. See Results.
Evaluators on a run plan
The run dialog shows the evaluators inherited from the test suites in scope, and Add evaluator adds one to the plan itself. A plan evaluator reads the conversation and the trace only, because a plan may cover scenarios from suites with different fields. Use it for a check that applies to every scenario, such as a PII scanner.From the CLI and the API
--evaluator infers the mappings the way the suite editor does. --evaluators-json takes the full attachment list, for a tool call mapping:
evaluators on POST /api/v1/test-suites, PATCH /api/v1/test-suites/{id} and config.evaluators on POST /api/v1/run-plans/run. GET /api/simulation-runs/{scenarioRunId} answers with results.evaluations. See the Test Suites API reference and the Run Plans API reference. The Python and TypeScript SDKs take fields and evaluators on test_suites.create / testSuites.create, test_suites.update / testSuites.update and on run_plans.run / runPlans.run; the MCP tools platform_create_test_suite, platform_update_test_suite and platform_run_plan take the same.