Get started with LangWatch Skills in seconds: Set up evals, scenario tests, and tracing just by asking your AI coding assistant.
Traces
Prompts
Annotations
Automations
Scenarios
Evaluators
- Overview
- POSTExact Match Evaluator
- POSTLLM Answer Match
- POSTBLEU Score
- POSTLLM Factual Match
- POSTROUGE Score
- POSTSQL Query Equivalence
- POSTLLM-as-a-Judge Boolean Evaluator
- POSTLLM-as-a-Judge Category Evaluator
- POSTLLM-as-a-Judge Score Evaluator
- POSTRubrics Based Scoring
- POSTRagas Answer Correctness
- POSTRagas Answer Relevancy
- POSTRagas Context Precision
- POSTRagas Context Recall
- POSTRagas Context Relevancy
- POSTRagas Context Utilization
- POSTRagas Faithfulness
- POSTRagas Faithfulness
- POSTRagas Response Context Precision
- POSTRagas Response Context Recall
- POSTRagas Response Relevancy
- POSTContext F1
- POSTContext Precision
- POSTContext Recall
- POSTAzure Content Safety
- POSTAzure Jailbreak Detection
- POSTAzure Prompt Shield
- POSTOpenAI Moderation
- POSTPresidio PII Detection
- POSTCustom Basic Evaluator
- POSTCompetitor Blocklist
- POSTCompetitor Allowlist Check
- POSTCompetitor LLM Check
- POSTOff Topic Evaluator
- POSTQuery Resolution
- POSTSemantic Similarity Evaluator
- POSTSummarization Score
- POSTValid Format Evaluator
- POSTLingua Language Detection
Saved Evaluators
import langwatch
df = langwatch.datasets.get_dataset("dataset-id").to_pandas()
experiment = langwatch.experiment.init("my-experiment")
for index, row in experiment.loop(df.iterrows()):
# your execution code here
experiment.evaluate(
"ragas/response_relevancy",
index=index,
data={
"input": row["input"],
"output": output,
},
settings={}
)
[
{
"status": "processed",
"score": 123,
"passed": true,
"label": "<string>",
"details": "<string>",
"cost": {
"currency": "<string>",
"amount": 123
}
}
]Evaluators
Ragas Response Relevancy
Evaluates how pertinent the generated answer is to the given prompt. Higher scores indicate better relevancy.
POST
/
ragas
/
response_relevancy
/
evaluate
import langwatch
df = langwatch.datasets.get_dataset("dataset-id").to_pandas()
experiment = langwatch.experiment.init("my-experiment")
for index, row in experiment.loop(df.iterrows()):
# your execution code here
experiment.evaluate(
"ragas/response_relevancy",
index=index,
data={
"input": row["input"],
"output": output,
},
settings={}
)
[
{
"status": "processed",
"score": 123,
"passed": true,
"label": "<string>",
"details": "<string>",
"cost": {
"currency": "<string>",
"amount": 123
}
}
]Documentation Index
Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
Use this file to discover all available pages before exploring further.
Authorizations
API key for authentication
Body
application/json
Response
Successful evaluation
Was this page helpful?
⌘I