How to start evals when you have no dataset
Two ways to get your first evals running this week: build a small dataset from the documents you already have, or start from your users' real conversations. Step by step, with examples.
Manouk Draisma · · Updated · LLM Evals
Teams often hold off on evals because they think they need a big, labeled dataset first. They don't. A first eval needs a handful of cases you can read and a clear idea of what a good answer looks like, and you can have both by the end of the day.
How to start evals depends on where your agent is today:
- No users yet. Your agent is in development or a closed pilot, so there's no real traffic to learn from. You start from what you already wrote down: specs, docs, FAQs and support tickets.
- Real users already. People talk to your agent every day. You start from their conversations.
Both paths end in the same place: a small dataset and a few scenario tests that grow as you learn.
Small and readable beats big and perfect
When teams ask our CTO, Rogério Chaves, how much data they need, he tells them this:
"We do not want 10,000 examples. Quite the opposite. We want like 10 scenarios. They're not a lot, but you can fully understand what's going on."
A small set you can read tells you why the agent failed. A big generated set mostly gives you a percentage. And a dataset written before launch can only hold the cases you already imagined, so don't spend weeks polishing it. Get a first version, run it, and let the failures tell you what to add.
Path 1: no traffic yet, start from your documents
How your agent should behave is already written down somewhere, spread across your documents.
Step 1: Collect what you already wrote
Good material for a first dataset:
- The product requirements document (PRD) or spec. Every "the agent must" and "the agent never" line is a test case.
- Help center pages and FAQs. Real questions with approved answers.
- Policies. Refunds, returns, what the agent may promise and what goes to a person.
- Support tickets and emails your team answered before the agent existed.
- Meeting transcripts with the domain experts who know what a good answer is.
- Your git history. Every prompt fix was a bug someone found, so each one is a test you already paid for.
Put them in one folder in your repo, say docs/eval-sources/, and remove personal data first.
Step 2: Let your coding agent build the first dataset
You don't have to write the rows by hand. Our datasets skill turns Claude Code, Cursor or any other coding agent into an evaluation engineer. It reads your code, your prompts and the documents you point it to, asks you a few questions, and builds the dataset with you.
The skill first proposes a plan: the columns, the kinds of questions to cover and where each one comes from. Then it shows you a preview of a few rows to approve, builds the rest and uploads the dataset to LangWatch.
A good preview for a webshop support agent looks like this:
| input | expected_output |
|---|---|
| hi, order 48213 still not here, its been 9 days | Looks up the order and gives the carrier status and the expected date. No refund promise before checking. |
| can i return shoes i wore once outside | Worn items can't be returned. Offers the warranty route if the shoes are faulty. |
| charged twice?? | Asks for the order number, checks both charges, and explains that one is a hold that drops off. |
| I want to talk to a human | Hands off to a person right away, with the conversation so far. |
Look at the inputs. Real users type in lowercase, skip words and get annoyed. The skill is built to write rows like that, because a dataset of clean textbook questions passes every test and teaches you nothing.
Our CTO has one warning on generated data: "It's easy now with Claude for you to just generate the dataset with 100 entries, but if it's not representative of what people are actually doing, then it doesn't matter." So read the preview the way a user would, and push back on anything that sounds made up.
If your agent answers from a knowledge base, there's a recipe for that too. npx skills add langwatch/skills/recipes/generate-rag-dataset builds question and answer pairs straight from your documents, with the passage each answer should come from.
Step 3: Run your first eval on it
Open the dataset and press Run experiment. Pick your prompt or agent, then add an evaluator to score each answer. With expected answers in the dataset, LLM Answer Match compares each reply with the expected output. Without them, an LLM-as-a-judge evaluator scores each reply against criteria you write, like these:
The answer follows the refund policy.
It never promises a refund before the order is checked.
It never makes up a delivery date.
Read every failing row. A failure is either a real bug in the agent or a bad row in the dataset, and at this stage both are worth finding.
Step 4: Write scenarios for the conversations that matter
Some failures only show up over several turns, like the agent that gives in on the third ask or forgets to confirm before it acts. For those, write a scenario: a situation, plus the criteria the conversation has to meet. A simulated user plays the situation against your agent with different words on every run, and a judge checks each criterion.
Your spec is the best source for them. "If you have the PRD already, that is perfect," Rogério tells teams. Generate the scenarios from it, run them, and look at which ones fail: "As you execute, you see which scenarios are failing the most, and that will tell you what you should investigate first."
The scenarios skill writes them from the same documents:
This one comes from a refund policy: a customer charged twice, and three things the agent must do.

And here's a run of it, with the simulated user, the agent's replies and the judge's reasoning for each criterion:

Start with a handful. Rogério's advice to a team that had just connected their coding agent: "Some people go crazy and create a lot of scenarios. I would start with very few, just so you really understand what's going on."
Our guide on how to test AI agents goes deeper on writing them. If you'd rather not touch code, Langy, the assistant inside LangWatch, writes a scenario from text you paste in, such as a section of your spec or notes from a call. See test your agent from the platform.
Step 5: Let real users in, with tracing on
Your documents only hold what you already know. As soon as you safely can, let a small group of real users in and record what happens. That's where Path 2 begins.
Path 2: real users already, start from their conversations
If people already use your agent, your best dataset is sitting in their conversations. You only need to record them and find the ones that went wrong.
Step 1: Record every conversation as a trace
A trace is the full record of one request to your agent: what the user asked, what the agent answered, and every step in between, like the tools it called, the documents it looked up, the cost and the time it took. Traces from the same conversation are grouped, so you can replay anything a user went through.
Add the LangWatch SDK for Python, TypeScript or Go, or send plain OpenTelemetry. Or install the tracing skill with npx skills add langwatch/skills/tracing and ask your coding agent to instrument your code.
A week of traffic is enough to start, even from a beta with a handful of users.
Step 2: Ask a plain question of every conversation
With traces coming in, ask them questions in plain words. An Instant Eval judges a question like "the customer sounds frustrated" against every conversation in a time window, in about a minute for 10,000 of them. Try it on a hundred first:
langwatch instant-eval run "the customer sounds frustrated" --target threads --last 7d --limit 100
It answers with the conversations that match, and how sure the judge is about each one:

Read what the judge saw with langwatch instant-eval sample. When it flags the wrong things, sharpen the question with --criteria, then run it on the whole month. Good first questions are the ones you'd act on:
- Did the user leave without an answer?
- Did the agent hand off to a person when it didn't need to?
- Did the agent promise something our policy doesn't allow?
- Did the agent call the order tool before answering about an order?
Step 3: Turn what you find into a dataset
Open the matching conversations in the Trace Explorer, select the ones that went wrong and press Add to dataset. You pick which parts of the trace go into which column. See datasets.

This is easier than writing cases up front. Our CTO explains why: "If you ask people up front, they're not that good at it. But they are good at spotting an issue and saying 'no, that looks wrong'."
When you're not sure what the right answer was, send the traces to an annotation queue instead. A domain expert reviews each one and corrects the output, and the corrected traces go to the dataset as the expected answers.
The datasets skill from Path 1 works here too. With traces in your project, it reads them first and writes new rows in your users' own words.
Step 4: Make the worst ones scenario tests
When a failure is about how the whole conversation went, turn it into a scenario, the same way as in Path 1. Rogério describes the loop like this:
"You take a problem that happened in production, reproduce it as a scenario, and see the agent failing. Then you push a fix: sometimes changing the prompt, sometimes new tools or skills, and that makes the scenario pass. After you run this loop a few times, you have accumulated a bunch of test cases that you can run on your CI."
Step 5: Keep the dataset growing
Put your best question on live traffic as a monitor, and an automation adds every trace it fails to the dataset as it comes in. From then on, new failures land in the dataset without anyone copying them over.
The data flywheel, two years later
In 2024 we wrote about the data flywheel: use what your AI does in production to make it better, which brings more usage, which shows you new cases to fix. The idea still holds. What changed is that each step is now one action instead of a project.
| Step of the loop | In 2024 | In LangWatch today |
|---|---|---|
| Get a first dataset | Write examples in a spreadsheet | A coding agent skill drafts it from your docs, code and traces |
| Find what goes wrong | Read traces by hand, or sample a few hundred | Ask a plain question of every conversation with Instant Evals |
| Keep the examples | Export to a spreadsheet | Add traces to a dataset, or let an automation do it |
| Get the right answer | Ask an expert over chat | Annotation queues with corrected outputs |
| Check the fix | Rerun a few examples by hand | Experiments on the dataset, scenario tests in CI |
The discovery part has no shortcut, because the failures your agent meets are specific to your users and your domain.
Put this into production with LangWatch.
Trace your agents, run evaluations, and turn failures into repeatable tests.
Frequently asked questions
- How many examples do I need to start LLM evals?
- Fewer than you think. A handful of cases you can read and understand beats thousands of generated rows. Start with the five to ten situations that matter most to your business, and grow the set from real failures.
- Should I use synthetic data to start evals?
- Yes, when you have no users yet, as long as it looks like real use. Generate it from your own documents, code and support history, read every row, and drop anything that sounds made up. Once real users arrive, move your evals onto their conversations.
- What if my agent isn't in production yet?
- Build a first dataset and a few scenario tests from your spec, docs, FAQs and support tickets, and let a coding agent skill do the writing. Run them on every change. Then let a small group of real users in with tracing on, and add what goes wrong.
- What is a trace?
- A trace is the full record of one request to your AI agent: the user's message, the reply, and every step in between, such as tool calls, document lookups, cost and time. Evals on real traffic start from traces.
- What is an eval cold start?
- It's the loop teams get stuck in at the start: to ship the agent you want evals, to write good evals you want real data, and to get real data you have to ship. The way out is a small first dataset from what you already know, then growing it from real conversations.