Experiments Overview

​

What is an Experiment?

1. Dataset - A collection of test cases with inputs (and optionally expected outputs)
2. Target - What you’re testing: a prompt, model, API endpoint, or custom code
3. Evaluators - Scoring functions that assess output quality

​

When to Use Experiments

• Before deploying - Validate prompt changes don’t regress quality
• Comparing options - Test different models, prompts, or configurations side-by-side
• CI/CD gates - Automatically block deployments that fail quality thresholds
• Benchmarking - Track quality metrics over time across experiment runs

​

Getting Started

​

Quick Example

import langwatch

# Load your dataset
df = langwatch.dataset.get_dataset("my-dataset").to_pandas()

# Initialize experiment
evaluation = langwatch.experiment.init("prompt-v2-test")

# Run through dataset
for idx, row in evaluation.loop(df.iterrows()):
    # Execute your LLM
    response = my_llm(row["input"])

    # Run evaluators
    evaluation.evaluate(
        "ragas/faithfulness",
        index=idx,
        data={
            "input": row["input"],
            "output": response,
            "contexts": row["contexts"],
        },
    )

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();

// Load dataset
const dataset = await langwatch.datasets.get("my-dataset");

// Initialize experiment
const evaluation = await langwatch.experiments.init("prompt-v2-test");

// Run through dataset
await evaluation.run(
  dataset.entries.map(e => e.entry),
  async ({ item, index }) => {
    // Execute your LLM
    const response = await myLLM(item.input);

    // Run evaluators
    await evaluation.evaluate("ragas/faithfulness", {
      index,
      data: {
        input: item.input,
        output: response,
        contexts: item.contexts,
      },
    });
  },
  { concurrency: 4 }
);

​

Experiment Results

• Compare runs - See how different configurations perform side-by-side
• Drill into failures - Inspect individual test cases that scored poorly
• Track trends - Monitor quality metrics across experiment runs over time
• Export data - Download results for further analysis

​

CI/CD Integration

# GitHub Actions example
- name: Run quality experiments
  run: |
    python -c "
    import langwatch
    result = langwatch.experiment.run('my-experiment')
    result.print_summary()
    "
  env:
    LANGWATCH_API_KEY: ${{ secrets.LANGWATCH_API_KEY }}

​

Next Steps