> ## Documentation Index > Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details. # LLM-as-a-Judge: score outputs on criteria without expected answers > Score outputs against written criteria with an LLM-as-a-Judge evaluator when your dataset has inputs but no expected answers, and read the reason behind each verdict. Creative, advisory and open-ended tasks have no single correct answer, so an exact comparison does not work. An LLM-as-a-Judge evaluator reads the input and the output, applies the criteria you write, and returns a verdict with its reasoning. This guide builds an experiment for a business coaching assistant, where the dataset has questions from users and no expected answers. ## Prerequisites * A dataset with an `input` column. See [Datasets](/docs/datasets/overview). * A model provider configured in Settings, so the prompt column and the judge can call a model. ## Pick the judge type LangWatch ships three LLM-as-a-Judge evaluators. All three read the same inputs (`input`, `output` and, when you map it, `contexts`) and apply the criteria you write in the evaluator's prompt. | Evaluator | Returns | Use it when | | ------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | **LLM-as-a-Judge Boolean Evaluator** | Pass or fail | The output either meets the criteria or it does not. | | **LLM-as-a-Judge Score Evaluator** | A number on the scale your prompt defines. The default prompt asks for 0.0 to 1.0. | You want a graded verdict, for example how satisfied the user is likely to be. | | **LLM-as-a-Judge Category Evaluator** | One of the categories you define, each with a name and a description | You want to sort outputs into buckets, for example `on_topic`, `off_topic`, `refusal`. | ## Steps Open [Experiments](https://app.langwatch.ai/@project/experiments), click **New Experiment**, then **Create Experiment**. Under **Datasets**, click **Add** and pick **Select existing dataset**. Under **Prompts or Agents**, click **Add**, pick **Prompt**, then **New Prompt**. Pick the model, write the coaching assistant's system prompt, check that `input` maps to the dataset's `input` column under **Variables**, and click **Apply**. On the prompt column, click **Add evaluator**, then **New Evaluator**. Pick the **LLM as Judge** category and **LLM-as-a-Judge Boolean Evaluator**. Name it, pick the judge model, and replace the default prompt with your criteria. Write the criteria as a checklist the judge can apply to one answer at a time. For example: ``` You are evaluating a business coaching assistant. The output passes only if all of these are true: 1. It answers the question the user asked, not a different one. 2. It gives at least one concrete next step the user can take this week. 3. It does not invent numbers, laws or company facts. 4. It stays under 200 words. Fail the output if any criterion is not met, and name the criterion that failed. ``` The LLM-as-a-Judge Boolean evaluator editor with the criteria prompt In **Variables**, map `input` to the dataset's `input` column and `output` to the prompt column's output. Click **Create Evaluator**. If a required field stays unmapped, the workbench opens the mapping editor. Map the field there and click **Apply**. Click **Run** in the header. Each row fills with the assistant's answer and the judge's verdict. Click an evaluator chip to open its **Result** block. **Status** is the verdict and **Details** holds the judge's reasoning, including the criterion that failed. Read the failed rows first: a reason that names the same criterion on many rows points at what to change in the prompt. An evaluator chip popover with the score, label, status and the judge's reasoning in Details ## Calibrate the judge A judge is a model with a prompt, so check it before you trust its pass rate. * Read ten verdicts by hand. If you disagree with the judge on more than one or two, tighten the criteria: replace adjectives such as "helpful" with observable checks such as "gives a next step". * Keep the judge model separate from the model under test when you can, so the same blind spots do not score themselves. * Reuse the evaluator. It is saved under [Evaluators](https://app.langwatch.ai/@project/evaluators) and you can attach the same one to an online evaluation to score production traffic with the same criteria. See [Online Evaluation](/docs/evaluations/online-evaluation/overview). ## Iterate and compare runs Change the prompt, click **Run** again, then open **Results** and tick both runs in the **Experiment Runs** sidebar to see them side by side. The pass rate tells you whether the change helped; the reasons on the rows that flipped tell you why. **Also check:** [Answer correctness](/docs/evaluations/experiments/ui/answer-correctness) when your dataset has expected outputs, and [Comparison](/docs/evaluations/experiments/ui/pairwise-compare) to let a judge pick the best of two or more prompts per row instead of scoring each one alone.