Skip to main content
LLMs are non-deterministic systems that generate responses to open-ended questions. While this flexibility is powerful, it makes quality assurance challenging, as identical inputs may produce varying outputs. This variability can reduce confidence in production deployments and lead to unscalable testing practices that rely on subjective assessments. A systematic approach to measuring and improving LLM application quality requires two components: a dataset with representative test examples to evaluate your application, and well-defined quality metrics. These metrics can be strict (e.g., golden answers, retrieval metrics) or flexible (e.g., task completion, defined criteria and style guidelines). This guide explores in-depth use cases for experiments and demonstrates how to implement LLM evaluations using LangWatch.

​
Evaluating if the LLM is generating the right answers

Consider scenarios where a correct answer exists for your LLM given a set of example questions. This applies when you have internal documents for answer generation or when building a customer support agent that must provide accurate responses. This example demonstrates how to evaluate a customer support agent.
  1. Navigate to the experiments page and click “New Experiment”:
  1. Choose Experiment:
  1. Select a sample dataset. You can generate a new dataset with AI or use the provided Customer Support Agent example dataset: Download Dataset
  2. Choose “Upload CSV” and select the dataset file:
  1. Save the dataset. You should see all 200 examples displayed:
  1. Click “Next”. Select an executor to run your examples. This can be an API endpoint from your application, a Python code block, or a prompt.
For this example, select “Create a prompt”:
  1. Paste the following sample Customer Service prompt:
  1. Select the LLM to execute this evaluation. This example uses gemini-2.0-flash-lite:
  1. Select an evaluator based on your use case. Since this example has expected answers beforehand, select “Expected Answer Evaluation”:
  1. Select “LLM Answer Match”. This evaluator uses an LLM to compare the generated answer (output) with the gold standard answer (expected_output) to verify semantic equivalence, even when phrased differently:
  1. Click “Next”. Provide a name for your evaluation and click “Run Evaluation”:
  1. Review the results. This evaluation achieved a 95% pass rate, with only a few answers requiring refinement.
You can now iterate on the prompt to address the failing questions. After making adjustments, run the evaluation again to ensure no regressions occur in previously passing examples. The version marker allows you to compare performance across different iterations and identify the best-performing version. This completes your first experiment. You can now integrate this versioned prompt into your application using LangWatch APIs. Maintain this experiment in LangWatch to validate that future versions of prompts, datasets, or LLM models meet or exceed current performance standards before deploying to production.