Answer Correctness: check LLM answers against the expected output

This guide builds an experiment for a customer support assistant where every question has a known correct answer. You run the questions through a prompt, score each answer against the expected output, and read the pass rate.

Prerequisites

A dataset with an input column and an expected_output column. Create one from a CSV on the Datasets page, or create it inside the workbench in step 2. See Datasets.
A model provider configured in Settings, so the prompt column and the LLM Answer Match evaluator can call a model.

Steps

Create the experiment

Open Experiments, click New Experiment, then Create Experiment. The workbench opens. Click the generated name in the header and rename it, for example Support answer correctness.

Add the dataset

Under Datasets, click Add and pick Select existing dataset. To start from a file instead, pick Upload CSV. To type the rows in, pick Create new and add the input and expected_output columns.

The Datasets Add menu with Select existing dataset, Upload CSV and Create new

Add a prompt column

Under Prompts or Agents, click Add and pick Prompt, then New Prompt. In the editor, pick the model and write the system prompt. For example:

You are the customer support assistant for ACME.
Answer only from the ACME policies below. If the answer is not in the policies, say that you do not know and refer the customer to support@acme.example.

Policies:
- Returns are accepted within 30 days of purchase, in the original packaging.
- Standard shipping takes 3 to 5 business days.
- Orders can be changed or cancelled until they ship.

In the Variables section, check that input maps to the dataset’s input column, so each row’s question reaches the model. Click Apply.

The prompt editor with the model selector and the Variables section mapping input to the dataset column

Add the evaluator

On the prompt column, click Add evaluator, then New Evaluator. Pick the Expected Answer category.

The Expected Answer category with Exact Match Evaluator, LLM Answer Match, LLM Factual Match, ROUGE Score and BLEU Score

Choose one of:

Evaluator	Use it when
Exact Match	The answer must match the expected output character for character. Case, surrounding whitespace and punctuation are ignored by default; each is a setting.
LLM Answer Match	The answer may be worded differently and still be correct. A model compares the answer with the expected output and passes it when the meaning matches.

Name the evaluator. In Variables, map output to the prompt column’s output, expected_output to the dataset’s expected_output column and input to the dataset’s input column, then click Create Evaluator. If a required field stays unmapped, the workbench opens the mapping editor. Map the field there and click Apply.

The LLM Answer Match editor with output mapped to the prompt output and expected_output to the dataset column

Run

Click Run in the header. Each cell fills with the model’s answer and the evaluator chip on each row turns into a pass or a fail. To rerun a single row, use the button on that cell.

Read the results

Hover over the summary in the column header. Pass Rate is the share of rows the evaluator passed. Click a failed chip to read the evaluator’s details: for LLM Answer Match, the reason the model gave for the mismatch.

The column summary with the pass rate and the per-evaluator breakdown

Iterate and compare runs

Edit the prompt to fix the failing rows and click Run again. Every run is kept. Click Results in the header and tick the previous run and the new one in the Experiment Runs sidebar to see both side by side, with the pass rate of each run in the chart and both outputs per row, and check that no row that passed before now fails.

Two experiment runs compared side by side on the results page

Each saved prompt version is what your application loads through the prompt library, so the version that wins the comparison is the one you ship.

Common failures

Symptom	Cause
The run button reads Configure missing mappings first	An evaluator field or a prompt variable has no source. Open the editor and map the field marked Required.
Every evaluator chip shows an error	The evaluator’s model has no provider key. Add one under Settings, Model Providers, or change the evaluator’s model.
A cell shows No output yet	The column did not run for that row. Click the cell’s run button.

Also check: LLM-as-a-judge for datasets without expected outputs, and Run experiments from CI/CD to trigger this experiment from a pipeline.

Get Started

Agent Testing

Observability

Evaluations

Prompt Management

Platform

Examples & Cookbooks

Help

Answer Correctness: check LLM answers against the expected output

Prerequisites

Steps

Iterate and compare runs

Common failures

​Prerequisites

​Steps

​Iterate and compare runs

​Common failures

Prerequisites

Steps

Iterate and compare runs

Common failures