Multimodal Evaluation: images, PDFs and vision models in experiments

The built-in LLM-as-a-Judge evaluators accept images as inputs and outputs. Any vision-capable model (GPT-5.2, Claude Sonnet, Gemini and others) reads the images and scores them against the criteria in your prompt. This covers common multimodal use cases:

Image generation quality: score outputs of image generation models
Document parsing: evaluate extracted metadata from PDFs and scanned documents
Content moderation: detect NSFW or low-quality uploaded images
Visual QA: evaluate answers to questions about images
Image comparison: compare generated outputs against reference images

Image support works with all three LLM-as-a-Judge evaluator types:

Boolean: pass/fail evaluation (e.g. “Is the generated image photorealistic?”)
Score: numeric score evaluation (e.g. “Rate image quality from 1-5”)
Category: classification evaluation (e.g. “Classify the image as: excellent, good, poor”)

See also:

Dataset Images: Setting up image columns in datasets
Saved Evaluators: Reuse evaluators via API

Supported Image Formats

Images can be provided in any of these formats:

Format	Example
Image URL	`https://example.com/photo.png`
Base64 data URI	`data:image/png;base64,iVBORw0KGgo...`
Markdown image	`![alt text](https://example.com/photo.png)`

Supported extensions: .png, .jpg, .jpeg, .gif, .webp, .svg, .bmp, .tiff

Image detection is strict by design, a field is treated as an image only when the entire value is an image URL or base64 string. Mixed text-and-image content is sent as plain text. This prevents unintended multipart content when a field happens to contain an image URL as part of a longer string.

Evaluating Images via UI

Step 1: Create a Dataset with Image Columns

Go to Experiments, click New Experiment, then Create Experiment
Under Datasets, click Add, then Create new
Add columns and set their type to image (URL) in the column type select
Paste image URLs or base64 data URIs into the cells. The grid renders them as thumbnails you can click to expand.

A dataset grid with an image column showing thumbnails — A dataset with an image column rendered as thumbnails

Step 2: Add an LLM-as-a-Judge Evaluator

On the prompt or agent column, click Add evaluator, then New Evaluator
Pick the LLM as Judge category and one of the Boolean, Score or Category evaluators
Choose a vision-capable model (e.g. gpt-5.2, claude-sonnet-4-5-20250929)
Write a prompt that references the image fields, and map dataset columns to the evaluator’s input, output and contexts variables

The evaluator automatically detects image values and sends them as multipart content to the vision model. No special configuration needed.

LangWatch experiments workbench showing image evaluation with LLM-as-a-Judge score evaluator — Image evaluation workbench, LLM-as-a-Judge scoring virtual try-on quality with three image columns mapped to evaluator variables

In this example, a virtual try-on pipeline is evaluated with three image columns:

original → mapped to contexts (the person’s photo)
request → mapped to input (the clothing item)
generated → mapped to output (the try-on result)

The LLM-as-a-Judge prompt instructs the model to evaluate all three images and score the quality of the generated output.

Step 3: Run and Iterate

Click Run in the header, or the run button on the column. The model receives all images as vision content and returns a score, a pass or fail, or a category, with its reasoning in the evaluator’s Details. Use this workflow to iterate on your evaluator prompt until you have reliable evaluation criteria, then save it for reuse across experiments and CI/CD pipelines.

Custom Workflow Evaluators for Complex Logic

For more advanced evaluation pipelines, you can create a Custom Workflow Evaluator in the Evaluators page. This gives you a visual workflow builder where you can chain multiple LLM nodes, add image variables to prompts, and build multi-step evaluation logic.

LangWatch custom workflow evaluator showing image variables in prompt template — Custom workflow evaluator with image variables mapped to prompt template fields

In the workflow builder:

Add image-typed variables to your prompt node inputs
Use {{ "{{variable_name}}" }} syntax to reference images in the prompt template
Map dataset columns to the image variables in the entry node
The workflow handles multipart content assembly automatically

This is useful when you need to split evaluation into multiple steps, use different models for different aspects, or combine vision evaluation with text-based checks.

Evaluating Images via SDK

For programmatic evaluation from notebooks or CI/CD, use the Python or TypeScript SDK with a saved evaluator.

Using a Saved Evaluator

After iterating on your evaluator in the UI, save it and call it from code:

import langwatch

df = langwatch.dataset.get_dataset("my-image-dataset").to_pandas()

experiment = langwatch.experiment.init("image-quality-evaluation")

for index, row in experiment.loop(df.iterrows()):
    # Use your saved image evaluator
    experiment.evaluate(
        "evaluators/image-quality-scorer",  # Your saved evaluator slug
        index=index,
        data={
            "input": row["request_image"],      # Image URL or base64
            "output": row["generated_image"],    # Image URL or base64
            "contexts": [row["original_photo"]], # List of context images
        },
    )

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();

const dataset = await langwatch.datasets.get("my-image-dataset");
const experiment = await langwatch.experiments.init("image-quality-evaluation");

await experiment.run(
  dataset.entries.map((e) => e.entry),
  async ({ item, index }) => {
    // Use your saved image evaluator
    await experiment.evaluate("evaluators/image-quality-scorer", {
      index,
      data: {
        input: item.request_image,       // Image URL or base64
        output: item.generated_image,    // Image URL or base64
        contexts: [item.original_photo], // List of context images
      },
    });
  },
  { concurrency: 4 }
);

Custom Scoring with Vision Models

You can also call vision models directly and log custom scores:

import langwatch
import litellm

experiment = langwatch.experiment.init("custom-image-evaluation")

for index, row in experiment.loop(df.iterrows()):
    # Call a vision model directly
    response = litellm.completion(
        model="gpt-5-mini",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Rate this generated image quality from 1 to 5. Return only the number."},
                {"type": "image_url", "image_url": {"url": row["generated_image"]}},
            ],
        }],
    )

    score = int(response.choices[0].message.content.strip())

    experiment.log(
        "image_quality",
        index=index,
        data={"output": row["generated_image"]},
        score=score / 5.0,
        passed=score >= 3,
        details=f"Image quality score: {score}/5",
    )

Evaluating Document Parsing (PDFs)

Multimodal evaluation also covers document-based pipelines. Here is an example of evaluating a PDF parsing pipeline that extracts metadata from academic papers:

import langwatch
import pandas as pd
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.base import elements_to_text
import dspy

dspy.configure(lm=dspy.LM("openai/gpt-5-mini"))

# Dataset of PDFs with ground truth metadata
df = pd.DataFrame([
    {
        "file": "paper1.pdf",
        "expected_title": "Vibe Coding vs. Agentic Coding",
        "expected_authors": "Ranjan Sapkota, Konstantinos I. Roumeliotis, Manoj Karkee",
    },
    # ... more rows
])

@langwatch.trace()
def extract_pdf_info(filename):
    langwatch.get_current_trace().autotrack_dspy()
    elements = partition_pdf(filename=filename)
    pdf = elements_to_text(elements=elements)
    return dspy.Predict(
        "pdf -> title: str, author_names: str, github_link: Optional[str]"
    )(pdf=pdf)

# Run the evaluation
evaluation = langwatch.experiment.init("pdf-parsing-evaluation")

for index, row in evaluation.loop(df.iterrows()):
    response = extract_pdf_info(row["file"])

    evaluation.log(
        "author_names_accuracy",
        index=index,
        passed=response.author_names == row["expected_authors"],
        details=f"Expected: {row['expected_authors']}, Got: {response.author_names}",
    )

Using Evaluators via API

Once you have a reliable image evaluator, you can call it directly via REST API for integration into any pipeline:

curl -X POST "https://app.langwatch.ai/api/evaluations/evaluators/image-quality-scorer/evaluate" \
     -H "X-Auth-Token: $LANGWATCH_API_KEY" \
     -H "Content-Type: application/json" \
     -d @- <<EOF
{
  "data": {
    "input": "https://example.com/clothing-item.jpg",
    "output": "https://example.com/tryon-result.jpg",
    "contexts": ["https://example.com/original-photo.jpg"]
  }
}
EOF

Base64 image payloads can be large. The evaluator API supports request bodies up to 30 MB. If you are working with many high-resolution images, prefer using image URLs over base64 encoding.

Model Compatibility

Image evaluation requires a vision-capable model. Any model supported by litellm with vision capabilities works, including:

Provider	Models
OpenAI	`gpt-4o`, `gpt-4o-mini`, `gpt-5.2`
Anthropic	`claude-sonnet-4-5-20250929`, `claude-opus-4-6`
Google	`gemini-2.0-flash`, `gemini-2.5-pro`

The evaluator sends every detected image as image content to the model you select, whether or not that model reads images. Pick a vision-capable model, or the provider can reject the request.

Get Started

Agent Testing

Observability

Evaluations

Prompt Management

Platform

Examples & Cookbooks

Help

Multimodal Evaluation: images, PDFs and vision models in experiments

Supported Image Formats

Evaluating Images via UI

Step 1: Create a Dataset with Image Columns

Step 2: Add an LLM-as-a-Judge Evaluator

Step 3: Run and Iterate

Custom Workflow Evaluators for Complex Logic

Evaluating Images via SDK

Using a Saved Evaluator

Custom Scoring with Vision Models

Evaluating Document Parsing (PDFs)

Using Evaluators via API

Model Compatibility

Next Steps

Dataset Images

Saved Evaluators

Experiments via SDK

CI/CD Integration

​Supported Image Formats

​Evaluating Images via UI

​Step 1: Create a Dataset with Image Columns

​Step 2: Add an LLM-as-a-Judge Evaluator

​Step 3: Run and Iterate

​Custom Workflow Evaluators for Complex Logic

​Evaluating Images via SDK

​Using a Saved Evaluator

​Custom Scoring with Vision Models

​Evaluating Document Parsing (PDFs)

​Using Evaluators via API

​Model Compatibility

​Next Steps

Dataset Images

Saved Evaluators

Experiments via SDK

CI/CD Integration

Supported Image Formats

Evaluating Images via UI

Step 1: Create a Dataset with Image Columns

Step 2: Add an LLM-as-a-Judge Evaluator

Step 3: Run and Iterate

Custom Workflow Evaluators for Complex Logic

Evaluating Images via SDK

Using a Saved Evaluator

Custom Scoring with Vision Models

Evaluating Document Parsing (PDFs)

Using Evaluators via API

Model Compatibility

Next Steps