Online Evaluation Overview

​

How It Works

User Request → Your LLM → Response → LangWatch Trace → Monitor → Score
                                                          ↓
                                              Dashboard & Alerts

1. Your application sends traces to LangWatch (via SDK integration)
2. Monitors evaluate incoming traces using your configured evaluators
3. Scores are recorded and displayed on dashboards
4. Optionally trigger alerts when scores drop below thresholds

​

When to Use Online Evaluation

​

Monitors vs Guardrails

​

Getting Started

​

Quick Setup

​

1. Ensure traces are being sent

import langwatch

@langwatch.trace()
def my_llm_app(user_input):
    # Your LLM logic here
    return response

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();
const trace = langwatch.getTrace();

// Your LLM logic here
trace.end();

​

2. Create a Monitor

1. Go to Online Evaluations in LangWatch
2. Click New Online Evaluation
3. Select Real-time evaluation (this creates a Monitor)
4. Choose “When a message arrives” as the trigger
5. Select evaluators (e.g., PII Detection, Faithfulness)
6. Configure any filters (optional)
7. Enable monitoring

​

3. View Results

• Traces - Individual trace scores visible in trace details
• Analytics - Aggregate metrics over time
• Alerts - Configure automations for low scores

​

Adding Scores via Code

import langwatch

@langwatch.trace()
def my_llm_app(user_input):
    response = generate_response(user_input)
    
    # Add a custom score
    langwatch.get_current_span().add_evaluation(
        name="response_quality",
        passed=True,
        score=0.95,
        details="High quality response"
    )
    
    return response

const trace = langwatch.getTrace();

// After generating response
trace.addEvaluation({
  name: "response_quality",
  passed: true,
  score: 0.95,
  details: "High quality response"
});

​

Available Evaluators

• Quality: Faithfulness, Answer Relevancy, Coherence
• Safety: PII Detection, Jailbreak Detection, Content Moderation
• RAG: Context Precision, Context Recall, Groundedness
• Custom: LLM-as-Judge with your own criteria

​

Next Steps