List of Evaluators

LangWatch offers a library of evaluators to check the quality and safety of your LLM apps.

How to use these evaluators:

Built-in Evaluators - Use directly in your code with the slug (e.g., ragas/faithfulness)
Saved Evaluators - Configure on the platform and reuse via evaluators/{slug}
Custom Scoring - Send your own evaluation scores

Evaluators API Reference

Full API documentation for running evaluations programmatically.

Evaluators List

Expected Answer

For when you have the golden answer and want to measure how correct the LLM gets it

Evaluator	Description
Exact Match Evaluator	Passes when the output matches the expected output character for character. Case, whitespace and punctuation handling are settings.
LLM Answer Match	Uses an LLM to check if the generated output answers a question correctly the same way as the expected output, even if their style is different.
LLM Factual Match	Uses an LLM to check that the facts stated in the output match the expected output.
BLEU Score	String similarity between the output and the expected output, from 0 to 1.
ROUGE Score	Overlap between the output and the expected output, from 0 to 1.

LLM as Judge

For when you don’t have a golden answer, but have a set of rules for another LLM to evaluate quality

Evaluator	Description
LLM-as-a-Judge Boolean Evaluator	Returns pass or fail against the criteria you write in the prompt.
LLM-as-a-Judge Score Evaluator	Returns a number on the scale your prompt defines.
LLM-as-a-Judge Category Evaluator	Returns one of the categories you define, each with a name and a description.
Rubrics Based Scoring	Scores the output against a rubric with a description for each score, typically from 1 to 5.
Comparison	Shows a judge every candidate output for the same row and picks the best one, or a tie. Used in experiments with two or more prompt or agent columns.

RAG Quality

For measuring the quality of your RAG, check for hallucinations with faithfulness and precision/recall

Evaluator	Description
Ragas Faithfulness	Measures how consistent the output is with the retrieved contexts. Higher scores mean fewer hallucinations.
Ragas Response Relevancy	Measures how relevant the output is to the input.
Ragas Response Context Precision	Measures how much of the retrieved context was needed for the output, using an LLM.
Ragas Response Context Recall	Measures how much of the expected output the retrieved contexts cover, using an LLM.
Context F1	Balances precision and recall of the retrieved contexts against the expected contexts, with string distance metrics.

Quality Aspects

For when you want to check the language, structure, style and other general quality metrics

Evaluator	Description
Valid Format Evaluator	Checks that the output is valid JSON, Markdown, Python, SQL or another format you pick. Works as a guardrail.
Language Detection	Detects the language of the input and the output, and passes when they match or when the output is in the language you expect. Works as a guardrail.
Sentiment Evaluator	Scores the sentiment of the text from -1.0 (very negative) to 1.0 (very positive), with a matching label.
Query Resolution	Checks with an LLM that every question the user asked in the conversation was answered.
Summarization Score	Measures how well the summary in the output captures the important information from the retrieved contexts.
SQL Query Equivalence	Checks with an LLM that the SQL query in the output returns the same results as the reference query, given the table schemas.

Safety

Check for PII, leaked secrets, prompt injection and harmful content, and keep the conversation on topic

Evaluator	Description
PII Detection	Detects personally identifiable information such as names, emails, phone numbers and credit card numbers, with Presidio. Works as a guardrail.
API Keys & Secrets Detection	Flags leaked credentials in the input or the output: provider and cloud API keys, tokens, private keys and database connection strings. Works as a guardrail.
Prompt Injection Detection	Detects prompt injection attempts in the input, with Azure Prompt Shield. Works as a guardrail.
Jailbreak Detection	Detects jailbreak attempts in the input, with Azure Content Safety. Works as a guardrail.
Content Safety	Detects hate, self-harm, sexual and violent content, with Azure Content Safety. Works as a guardrail.
Moderation	Flags harmful content with the OpenAI moderation API. Works as a guardrail.
Competitor Blocklist	Fails when the output mentions a competitor from your list. Works as a guardrail.
Competitor Allowlist Check	Uses an LLM to fail when the output mentions a company outside your allowlist. Works as a guardrail.
Competitor LLM Check	Uses an LLM with function calling to detect competitor mentions. Works as a guardrail.
Off Topic Evaluator	Uses an LLM to fail when the input is outside the topics you allow. Works as a guardrail.

API only

These evaluators run from the SDK and the REST API by their id and are not offered in the evaluator picker

Evaluator	Id
Custom Basic Evaluator	`langevals/basic`
Semantic Similarity Evaluator	`langevals/similarity`
Context Precision	`ragas/context_precision`
Context Recall	`ragas/context_recall`

Quick Start

Using a Built-in Evaluator

Use any evaluator from the list above directly in your code:

import langwatch

@langwatch.span()
def my_llm_step(user_input: str):
    output = my_llm(user_input)

    # Use any evaluator from the list above
    result = langwatch.evaluation.evaluate(
        "ragas/faithfulness",  # Evaluator slug from the list
        name="Faithfulness Check",
        data={
            "input": user_input,
            "output": output,
            "contexts": contexts,
        },
    )

    return output

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();

async function myLLMStep(userInput: string): Promise<string> {
  const output = await myLLM(userInput);

  // Use any evaluator from the list above
  const result = await langwatch.evaluations.evaluate("ragas/faithfulness", {
    name: "Faithfulness Check",
    data: {
      input: userInput,
      output: output,
      contexts: contexts,
    },
  });

  return output;
}

Learn more about using built-in evaluators →

Running Evaluations via UI

You can also run evaluations through the Experiments Workbench without writing code:

Get Started

Agent Testing

Observability

Evaluations

Prompt Management

Platform

Examples & Cookbooks

Help

List of Evaluators

Evaluators API Reference

Evaluators List

Expected Answer

LLM as Judge

RAG Quality

Quality Aspects

Safety

API only

Quick Start

Using a Built-in Evaluator

Running Evaluations via UI

Next Steps

Built-in Evaluators

Saved Evaluators

Custom Scoring

API Reference

Evaluators API Reference

​Evaluators List

​Expected Answer

​LLM as Judge

​RAG Quality

​Quality Aspects

​Safety

​API only

​Quick Start

​Using a Built-in Evaluator

​Running Evaluations via UI

​Next Steps

Built-in Evaluators

Saved Evaluators

Custom Scoring

API Reference

Evaluators List

Expected Answer

LLM as Judge

RAG Quality

Quality Aspects

Safety

API only

Quick Start

Using a Built-in Evaluator

Running Evaluations via UI

Next Steps