Online Evaluation Overview

​

How it works

User request → Your application → Response → Trace in LangWatch → Monitor → Score
                                                                       ↓
                                                     Trace drawer, analytics, automations

1. Your application sends traces to LangWatch through an SDK or OpenTelemetry.
2. Each monitor checks its preconditions and its sampling rate against the incoming trace.
3. The monitor runs its evaluator on the trace, or on the whole thread once the thread goes idle.
4. The result lands on the trace and in the evaluations analytics, where an automation can pick it up.

​

When to use online evaluation

​

Monitors and guardrails

​

Create a monitor

1. Open Online Evaluations and click New Online Evaluation.
2. Pick an evaluator category and an evaluator, name it and map its inputs to the trace.
3. Pick Trace Level or Thread Level, add preconditions and a sampling rate if you need them, and click Create Online Evaluation.

​

Add scores from your code

import langwatch

@langwatch.trace()
def my_llm_app(user_input):
    response = generate_response(user_input)

    langwatch.get_current_span().add_evaluation(
        name="response_quality",
        passed=True,
        score=0.95,
        details="High quality response",
    )

    return response

import { getLangWatchTracer } from "langwatch";

const tracer = getLangWatchTracer("my-app");

await tracer.withActiveSpan("my_llm_app", async (span) => {
  const response = await generateResponse(userInput);

  span.addEvaluation({
    name: "response_quality",
    passed: true,
    score: 0.95,
    details: "High quality response",
  });

  return response;
});

​

Next steps