Type Alias EvaluatorMapping

EvaluatorMapping:
| EvaluatorMappingLiteral
| ((state: ScenarioExecutionStateLike) => unknown)

Where one evaluator input reads its value from: a function of the scenario state, the same object a script step receives, sync or async; or a literal.

The function runs once the scenario has a verdict. What it returns is the input value. Returning nothing (undefined, null or an empty list) skips the evaluator; throwing reports an error result.

Example

mappings: {
  output: (state) => state.toolCalls("run_sql").last?.input,
  expected_output: (state) => state.field("golden_sql"),
  contexts: (state) => state.spans.filter((span) => span.attributes["langwatch.span.type"] === "rag"),
  language: "en",
}

Type Alias EvaluatorMapping

Example

Settings