Skip to main content
Creative, advisory and open-ended tasks have no single correct answer, so an exact comparison does not work. An LLM-as-a-Judge evaluator reads the input and the output, applies the criteria you write, and returns a verdict with its reasoning. This guide builds an experiment for a business coaching assistant, where the dataset has questions from users and no expected answers.

​
Prerequisites

  • A dataset with an input column. See Datasets.
  • A model provider configured in Settings, so the prompt column and the judge can call a model.

​
Pick the judge type

LangWatch ships three LLM-as-a-Judge evaluators. All three read the same inputs (input, output and, when you map it, contexts) and apply the criteria you write in the evaluator’s prompt.

​
Steps

1

Create the experiment and add the dataset

Open Experiments, click New Experiment, then Create Experiment. Under Datasets, click Add and pick Select existing dataset.
2

Add a prompt column

Under Prompts or Agents, click Add, pick Prompt, then New Prompt. Pick the model, write the coaching assistant’s system prompt, check that input maps to the dataset’s input column under Variables, and click Apply.
3

Create the judge

On the prompt column, click Add evaluator, then New Evaluator. Pick the LLM as Judge category and LLM-as-a-Judge Boolean Evaluator.Name it, pick the judge model, and replace the default prompt with your criteria. Write the criteria as a checklist the judge can apply to one answer at a time. For example:
The LLM-as-a-Judge Boolean evaluator editor with the criteria prompt
In Variables, map input to the dataset’s input column and output to the prompt column’s output. Click Create Evaluator. If a required field stays unmapped, the workbench opens the mapping editor. Map the field there and click Apply.
4

Run

Click Run in the header. Each row fills with the assistant’s answer and the judge’s verdict.
5

Read the reasons

Click an evaluator chip to open its Result block. Status is the verdict and Details holds the judge’s reasoning, including the criterion that failed. Read the failed rows first: a reason that names the same criterion on many rows points at what to change in the prompt.
An evaluator chip popover with the score, label, status and the judge's reasoning in Details

​
Calibrate the judge

A judge is a model with a prompt, so check it before you trust its pass rate.
  • Read ten verdicts by hand. If you disagree with the judge on more than one or two, tighten the criteria: replace adjectives such as “helpful” with observable checks such as “gives a next step”.
  • Keep the judge model separate from the model under test when you can, so the same blind spots do not score themselves.
  • Reuse the evaluator. It is saved under Evaluators and you can attach the same one to an online evaluation to score production traffic with the same criteria. See Online Evaluation.

​
Iterate and compare runs

Change the prompt, click Run again, then open Results and tick both runs in the Experiment Runs sidebar to see them side by side. The pass rate tells you whether the change helped; the reasons on the rows that flipped tell you why. Also check: Answer correctness when your dataset has expected outputs, and Comparison to let a judge pick the best of two or more prompts per row instead of scoring each one alone.
Last modified on September 6, 2026