Skip to main content
Let your agent set this up. Copy the evaluations prompt into your coding agent to get started automatically.
LangWatch makes it easy to run experiments from code. Just add a few lines to start tracking your experiments.

Quickstart

1. Install the SDK

2. Set your API Key

Be sure to login or create an account on the link that will be displayed, then provide your API key when prompted.

3. Start tracking

That’s it! Your evaluation metrics are now being tracked and visualized in LangWatch.
Evaluation Results Sample

Core Concepts

Evaluation Initialization

The evaluation is started by creating an evaluation session with a descriptive name:

Iterating over data

Use evaluation.loop() around your iterator so the entries are tracked:

Metrics logging

Track any metric you want with evaluation.log():

Comparing Multiple Targets

When comparing different models, prompts, or configurations, use targets to organize your results. Both SDKs provide a target(), withTarget() context that automatically captures latency and enables context inference.
Use evaluation.target() for automatic latency capture and context inference:
evaluation.target() automatically captures latency, creates isolated traces per target, and enables context inference so log() calls don’t need explicit target parameters. Use log_response() to store the model’s output.
Alternatively, use the target parameter directly with evaluation.log():

Target Registration

The first time you use a target name, it’s automatically registered with the provided metadata:
If you provide different metadata for the same target name, an error will be raised. Use a different target name if you want different configurations.

Metadata for Comparison

Target metadata is used for comparison charts in the LangWatch UI. You can group results by any metadata field:
In the LangWatch UI, you can then visualize how quality varies across temperature values.

Parallel Execution

LLM calls can be slow. Both SDKs support parallel execution to speed up your evaluations.
Use the built-in parallelization by putting the content of the loop in a function and submitting it:
By default, threads=4. Adjust based on your API rate limits and system resources.

Async-native mode

The default loop(), submit() path above already parallelises, each submitted task runs in a worker thread, so sync and async tasks both speed up with no extra work on your side. That’s the right choice for most users.Reach for aloop(), asubmit() only when your code is fully async-first and your task relies on async state whose identity is tied to one event loop. The threading path spins up a fresh event loop per worker, so those objects raise "Future attached to a different loop" on first use. aloop, asubmit keep every submitted task on the caller’s event loop, so that state stays valid across concurrent items.
Sync callables passed to asubmit are automatically offloaded to a worker thread so they don’t block the event loop for concurrent async siblings.

Built-in Evaluators

LangWatch provides a comprehensive suite of evaluation metrics out of the box.
Use evaluation.run() to leverage pre-built evaluators:
Browse our complete list of available evaluators including metrics for RAG quality, hallucination detection, safety, and more.

Pairwise Compare

langevals/pairwise_compare compares two candidate outputs for the same input and asks a judge model to pick the better one. This is useful when you want to compare two prompts, models, or agent configurations directly instead of scoring each one independently. The example below was verified against the live SDK path using evaluation.evaluate("langevals/pairwise_compare", ...).
swap_and_confirm performs two judge calls with A/B order reversed on the second pass. If the two calls disagree, the evaluator returns a tie. Set has_golden_answer to false when you want a pure head-to-head comparison without a reference answer. The SDK result also records the pairwise judge’s own evaluator cost and duration. In exported results, those appear under the evaluator columns for langevals/pairwise_compare; they are separate from include_metrics, which injects each candidate target’s cost or duration into the judge prompt. If you already have your own local pairwise judge, publish its result with evaluation.log(...) by sending a score, label, and details. Those custom results are tracked like any other SDK metric. The dedicated pairwise column visualization currently comes from the built-in Pairwise Compare evaluator configuration.

Complete Example

Tracing Your Pipeline

To get complete visibility into your LLM pipeline, add tracing to your functions:
Learn more in our Python Integration Guide.
With tracing enabled, you can click through from any evaluation result to see the complete execution trace, including all LLM calls, prompts, and intermediate steps.

Exporting Results to CSV

After running your evaluations, you can export results to CSV for further analysis in spreadsheet tools like Excel or Google Sheets.

How to Export

Click the Export to CSV button in the top-right corner of the evaluation results page to download a complete CSV file with all your data.

CSV Structure

The exported CSV contains comprehensive data organized by dataset rows and targets. Here’s the complete column structure:

Row Index

Dataset Columns

All columns from your input dataset are included with their original names.

Target Columns (per target)

For each target in your evaluation, the following columns are exported:

Evaluator Columns (per target, per evaluator)

For each evaluator applied to a target: For langevals/pairwise_compare, the evaluator cost and duration columns describe the pairwise judge run itself. When swap_and_confirm is enabled, that judge run may include two model calls.

Example CSV Output

For an evaluation comparing GPT-4 and Claude with a faithfulness evaluator:

Using the Data

The CSV export enables powerful analysis workflows:
Use spreadsheet filters to compare specific models or configurations:
  • Filter by {target}_model to analyze specific model performance
  • Sort by {target}_{evaluator}_score to find best/worst performing samples
  • Filter by {target}_error to identify failed executions
Calculate aggregate metrics across your evaluation:
  • Sum {target}_cost columns for total evaluation cost per model
  • Average {target}_duration_ms to compare response times
  • Identify outliers with high latency or cost
Analyze performance across different configurations:
  • Pivot tables by temperature, max_tokens, or custom metadata
  • Compare prompt versions side-by-side
  • Track improvements across iterations
Investigate problematic samples:
  • Filter rows where {target}_error is not empty
  • Cross-reference index with the UI for detailed inspection
  • Click through to traces using {target}_trace_id
All column headers are normalized to lowercase with spaces replaced by underscores for consistency and compatibility with data analysis tools.

Running in CI/CD

You can run SDK experiments in your CI/CD pipeline. The print_summary() method outputs a structured summary and exits with code 1 if any evaluations fail:
See CI/CD Integration for complete examples with GitHub Actions, GitLab CI, and more.

What’s Next?

CI/CD Integration

Run experiments in your CI/CD pipeline

View Evaluators

Explore all available evaluation metrics

Datasets

Learn about dataset management

View Examples

Check out example notebooks