Skip to main content
A monitor is an evaluator attached to your production traffic. Traces that match its preconditions and sampling rate are scored asynchronously, so the score never delays the response to your user. A Trace Level monitor scores each trace as it arrives; a Thread Level monitor waits for the thread to go idle first. This guide sets up an LLM-as-a-Judge monitor named Off-topic reply that flags replies that drift away from the customer’s question. The same steps apply to any evaluator, for example PII detection or jailbreak detection.

​
Prerequisites

  • Your application sends traces to LangWatch. See the integration quick start.
  • For evaluators that call a model, a model provider configured in Settings.

​
Create a monitor

1

Pick an evaluator

Open Online Evaluations and click New Online Evaluation. In the drawer, pick the Evaluation Level, then click Select Evaluator. Pick one of your saved evaluators, or click New Evaluator to open Choose Evaluator Category: Expected Answer, LLM as Judge, RAG Quality, Quality Aspects and Safety, plus Custom (Code) for a Python evaluator and Custom (from Workflow) for a workflow.Pick LLM as Judge, then LLM-as-a-Judge Boolean Evaluator.
The Choose Evaluator Category drawer
2

Configure the evaluator

Name the evaluator, pick the judge model and write the criteria in Prompt, for example: the reply passes only when it answers the customer’s question and stays on the topic of the customer’s order. A built-in evaluator such as PII Detection shows its own options here instead, for example the entity types to flag.Under Variables, map the evaluator’s fields to the trace. input maps to the trace input and output to the trace output by default. Click Create Evaluator.
3

Set the level, the filter and the sampling

Back in the New Online Evaluation drawer, the evaluator is selected.
The New Online Evaluation drawer with the evaluation level, the evaluator, the name, a precondition and the sampling rate
Click Create Online Evaluation. The monitor is active as soon as it is created.

​
Evaluate whole threads

A trace is one request and response. A thread is the whole conversation, grouped by the thread_id you send in the trace metadata. Some checks only make sense on the conversation: whether the user’s problem was resolved, whether the assistant stayed consistent across turns, whether the tone drifted. Pick Thread Level in the drawer. The evaluator’s variables now map to the Current Thread source, whose traces field holds every trace in the thread with its input and output.
The drawer with Thread Level selected and the Thread Idle Time select
Thread Idle Time sets how long the monitor waits after the last message before it scores the thread, from 1 to 30 minutes, 5 minutes by default. Without it the thread would be scored again on every new message. Pick Disabled - evaluate on every trace to score after each message anyway.

​
Read the results

On a trace, the Evals section of the trace drawer lists each monitor that ran with its verdict, score and the reason it returned, and the header shows an Eval chip per monitor. The trace table has an Evals column too, so you can filter and sort the Trace Explorer by the monitor’s result.
The Evals section of a trace drawer with the Off-topic reply verdict
The Online Evaluations list shows every monitor with its mode, its status and Performance, last 7 days: the pass rate or the average score with a trend against the previous week.
The Online Evaluations list with the active Off-topic reply monitor and its 7 day pass rate
View analytics in the row menu opens the evaluations analytics page. At the top, Overall Health counts the evaluation runs and Pass / Fail Trend plots them over time across every monitor. Below, each monitor has a pass and fail share, a bar chart of passed and failed evaluations per day and a Pass Rate Trend line. A score evaluator shows its average score instead of a pass rate.
The evaluations analytics page with the Off-topic reply monitor: the pass and fail share, the passed and failed evaluations per day and the Pass Rate Trend

​
Pause, edit and remove a monitor

The row menu on the list has Edit, Disable (or Enable on a paused monitor), Replicate to another project and Delete. A paused monitor keeps its history and stops scoring new traces.

​
Act on the results

  • To be notified when a monitor fails, create an automation on the evaluation result. See Automations.
  • To collect the flagged traces into a dataset, add an automation with the Add To Dataset action. See Build datasets from traces.
  • To block a request before the response reaches the user, run the evaluator as a guardrail from your code instead of as a monitor. See Guardrails.
Also check: Evaluators for the full list of evaluators a monitor can run, and Saved evaluators to reuse one evaluator across experiments and monitors.
Last modified on September 6, 2026