Documentation Index
Fetch the complete documentation index at: /docs/llms.txt
Use this file to discover all available pages before exploring further.
Get started with LangWatch Skills in seconds: Set up evals, scenario tests, and tracing just by asking your AI coding assistant.
Score your prompts and agents on a dataset before you ship, score production traffic as it arrives, and block unsafe responses before they reach the user.
| Goal | Use |
|---|---|
| Check a prompt change on a dataset before you deploy it | Experiments |
| Compare two prompts, models or agents on the same rows | Comparison |
| Gate a deployment on a pass rate from your pipeline | Run experiments from CI/CD |
| Score production traffic and follow the pass rate over time | Online Evaluation |
| Block a jailbreak or a policy violation before the response is sent | Guardrails |
| Get a Slack message or an email when a monitor fails | Online Evaluation with Automations |
| Score outputs by hand and collect the reviews | Annotations |
import langwatch
evaluation = langwatch.experiment.init("my-first-experiment")
for idx, row in evaluation.loop(dataset.iterrows()):
response = my_llm(row["input"])
evaluation.log("quality", index=idx, score=0.95)
import { LangWatch } from "langwatch";
const langwatch = new LangWatch();
const evaluation = await langwatch.experiments.init("my-first-experiment");
await evaluation.run(dataset, async ({ item, index }) => {
const response = await myLLM(item.input);
evaluation.log("quality", { index, score: 0.95 });
});
as_guardrail=True and check passed before you answer the user:
import langwatch
@langwatch.trace()
def my_llm_call(user_input):
guardrail = langwatch.evaluation.evaluate(
"azure/jailbreak",
name="Jailbreak Detection",
as_guardrail=True,
data={"input": user_input},
)
if not guardrail.passed:
return "I can't help with that request."
return call_llm(user_input)
Was this page helpful?