# How to test a voice AI agent before it takes real calls

Voice agent testing means a simulated caller phones your agent over real audio, with noise, interruptions and accents, and a judge passes or fails the call against your criteria. Run it on every change instead of listening to calls by hand.

*By Rogerio Chaves · November 3, 2026*

Canonical: https://langwatch.ai/blog/how-to-test-voice-ai-agents

![How to test a voice AI agent before it takes real calls](https://langwatch.ai/blog/how-to-test-voice-ai-agents-cover.webp)

**Short answer:** To test a voice AI agent, have a simulated caller phone it over real audio, with background noise, interruptions and the accents your callers have, and let a judge pass or fail the call against criteria you write. Turn real calls into scenarios, run them on every change, and listen only to the calls that fail.

Voice agent testing is the same idea as testing a chat agent, with a phone line in the middle. The code below uses [Scenario](https://langwatch.ai/scenario/), the open-source agent testing framework we build at LangWatch, which drives a call against the voice agent you already deployed.

## How to test voice AI agents before production?

Describe a real call as a scenario: who is calling, what they want, and what the agent must and must not do. A simulated caller speaks those lines as audio to your agent, over the same transport it uses in production, and listens to each reply before it answers. A judge reads the call and passes or fails each criterion with its reasoning.

The audio is what makes it a voice test. Run the same conversation as text and you test the prompt and the tools, which is worth doing too. The bugs that only voice has live in the speech-to-text, the turn-taking and the timing, so the test has to go through them.

| Step | What you do | What it catches |
|---|---|---|
| Write the scenario | One situation from a real call, plus criteria | The agent doing the wrong thing |
| Shape the caller | Voice, accent, noise, interruptions | Bugs that only show up in audio |
| Judge the call | Criteria on the transcript and the agent's traces, latency from the audio | Wrong answers, wrong tool calls, slow replies |
| Run on every change | CI, or a run from the platform | Regressions from a prompt, model or provider change |

## What breaks only in voice: interruptions, silence, accents, wrong numbers

A voice agent can give the right answer and still fail the call, in four ways a chat test never triggers:

1. **Interruptions.** The caller cuts in with "sorry, which day?" and the agent keeps reading its whole paragraph, or it stops and loses track of what it was saying.
2. **Silence and slow replies.** The agent waits too long after the caller finishes, so the caller says "hello?" into dead air, or it jumps in before they're done. Voice activity detection decides when a turn ends, and a pause in the middle of a sentence can fool it.
3. **Accents and bad lines.** A strong accent, a street behind the caller or a phone line that cuts the audio to a narrow band, and the speech-to-text hears something else. The agent then answers the question it heard.
4. **Wrong numbers.** Phone numbers, order ids and dates are where speech-to-text slips. "Fifteen" turns into "fifty", one digit goes missing, and the agent books the wrong slot with full confidence. The fix is to read the number back and wait for a yes, so that's a criterion worth writing.

Some failures come from the person who picks up. Take an outbound agent that calls carriers to offer freight loads. A common case is the contact in the system being the wrong person: "no, you need to talk to my colleague". The agent has to catch that, ask for the colleague's number and save it, instead of pitching the load to someone who can't take it.

## Simulated callers over real audio, and a judge on every call

Before release you have no real calls to grade, so you generate them. The simulated caller is an LLM with a text-to-speech voice. It improvises from the scenario description, so every run sounds a bit different, and its audio goes through effects like background noise or a phone line before your agent hears it.

Here is that case as a test, for a Pipecat bot. The shape comes from the [voice quick start](https://langwatch.ai/scenario/voice/getting-started/), and the caller settings from the [audio effects](https://langwatch.ai/scenario/voice/recipes/effects/) and [interruptions](https://langwatch.ai/scenario/voice/recipes/interrupt/) recipes:

```python
import os

import pytest
import scenario

BOT_WS_URL = os.environ.get("PIPECAT_BOT_URL", "ws://localhost:8765/stream")


@pytest.mark.agent_test
@pytest.mark.asyncio
@pytest.mark.timeout(300)
async def test_wrong_person_gives_a_colleagues_number():
    result = await scenario.run(
        name="wrong person, colleague's number",
        description=(
            "The agent calls a carrier to offer a load. The person who picks up "
            "is not the dispatcher. They say a colleague handles loads and give "
            "the colleague's mobile number once, quickly."
        ),
        agents=[
            # your deployed bot, over the WebSocket it already serves
            scenario.PipecatAgentAdapter(
                url=BOT_WS_URL,
                audio_format="mulaw",
                sample_rate=8000,
            ),
            scenario.UserSimulatorAgent(
                voice="openai/nova",
                persona="Busy, speaks in short sentences, cuts in when the agent talks too long.",
                interrupt_probability=0.5,
                audio_effects=[
                    scenario.effects.background_noise("street", volume=0.4),
                    scenario.effects.phone_quality(),
                ],
            ),
            scenario.JudgeAgent(criteria=[
                "The agent does not pitch the load to the person who picked up",
                "The agent asks for the colleague's name and number",
                "The agent reads the number back and waits for the caller to confirm it",
                "The agent stops talking when the caller cuts in",
            ]),
        ],
        script=[
            scenario.agent(),
            scenario.user(),
            scenario.proceed(turns=5),
            scenario.judge(),
        ],
    )

    if result.audio is not None:
        result.audio.save("recordings/wrong_person.wav")
    assert result.success
```

`interrupt_probability` makes the caller barge in on about half of the agent's turns. When you need one interruption at an exact spot, script it with `scenario.interrupt("Sorry, which day was that?")` right after a `scenario.user(...)` step. For accents, pick the caller's voice: an accent is a text-to-speech voice, so you set it with `voice=`, and there's no accent effect on top.

The judge reads the transcript of the call, plus your agent's traces when you trace the agent with LangWatch, so it checks a criterion about a tool call against the tool call the agent made. It reads text and can't hear a robotic tone, which is why the test saves the recording. When a run fails, play `recordings/wrong_person.wav` next to the judge's reasoning.

Timing comes from the audio itself. After a run, `result.latency` holds the time to the first byte of agent audio and the median and 95th-percentile response times across turns, so you can track them per scenario and fail the test above a budget you set from your own production calls:

```python
assert result.latency is not None
print(f"p95 response time: {result.latency.p95_response_time:.2f}s")
```

The same test works on other stacks by swapping the first agent. Scenario has [adapters](https://langwatch.ai/scenario/voice/choosing-an-adapter/) for ElevenLabs agents, Twilio phone numbers, OpenAI Realtime, Gemini Live and your own speech-to-text, LLM and text-to-speech pipeline, in Python and TypeScript. LiveKit, Vapi and raw WebRTC adapters are in preview, Python only. The [voice agent testing](https://langwatch.ai/voice-ai-agent-testing) page lists every stack we connect to.

If you'd rather not write code, the [voice agents](https://langwatch.ai/docs/agent-testing/voice-agents) page in LangWatch does the same from the platform. You add the agent by its ElevenLabs agent id or a phone number, write the scenario in the editor, and set the caller's voice, how often they interrupt (from 0% to 100%) and a phone line or background noise effect. Voice agents on the platform are in private beta, so on cloud you write to support to turn them on.

## Testing in languages your team doesn't speak

Many voice agents call in more languages than their team speaks. If your agent talks to customers in Polish and no one on the team speaks it, you can't check those calls by ear at all.

You don't need a Polish speaker to run a Polish scenario. You describe the caller as someone who speaks only Polish. The criteria can stay in English, like "the agent confirms the pickup date" or "the agent uses formal Polish", and the judge checks them against the Polish transcript. Pick a judge model you trust in that language, because the verdict is only as good as the model's Polish.

Then the native speaker's job shrinks to the failures. A simple habit: run the same scenarios in each language, compare how they do, and send the speaker only the calls that failed, with the judge's reasoning next to each one.

## Testing LiveKit and Pipecat voice agents

Both frameworks ship their own testing tools now. LiveKit Agents has [unit tests that run in text mode](https://docs.livekit.io/testing/overview/) and agent simulations that can also run over audio. Pipecat has [Pipecat Evals](https://docs.pipecat.ai/pipecat/evals/overview), with scripted and simulated scenarios in YAML files. If you're on one stack, they're a fine place to start.

Scenario fits when you want one test suite for every agent you run, chat and voice, in pytest or vitest, with the runs, recordings and pass rates in LangWatch next to your production traces. For Pipecat, `PipecatAgentAdapter` connects to the WebSocket your bot already serves. For LiveKit, the room adapter is in preview and Python only.

## From 200 calls a week by ear to tests on every change

A common starting point is a QA team that listens to a couple hundred calls a week, checking each one for issues and writing them down. That doesn't scale, and hiring more QAs is not the answer, so the testing has to be automated. Listening finds problems, but it can't tell them whether the next prompt change broke something.

Other teams do it the other way round and test on real people, with live calls in production. Each person on the other end gets a bad first call with the agent, and that contact is hard to win back.

Here's the loop I'd set up instead:

1. **Turn your worst calls into scenarios.** Take the calls your QA team flagged, or the ones you'd be embarrassed to replay, and write each one as a situation and its criteria.
2. **Run them on every change.** Voice scenarios run in your normal test runner, so they go into the same [CI job](https://langwatch.ai/scenario/basics/ci-cd-integration) as your other tests, set up the way our guide on [how to test AI agents](https://langwatch.ai/blog/how-to-test-ai-agents) shows. Each run takes about 30 to 120 seconds, so give the job a generous timeout and run the full set on prompt, model and provider changes.
3. **Watch the production calls.** Trace the agent with LangWatch, which [plays the audio back on the trace](https://langwatch.ai/docs/integration/python/tutorials/capturing-audio), and add [monitors](https://langwatch.ai/docs/evaluations/online-evaluation/overview) that score each conversation after it ends. A judge that checks "the caller had to repeat themselves" flags the calls worth a listen.
4. **Feed the failures back.** Every bad real call becomes a new scenario, so the same mistake fails a test before the next release instead of reaching another carrier.

Calls cost money, so plan for it. Every simulated call uses your voice provider's minutes, ElevenLabs or Twilio, plus OpenAI audio tokens for the caller. A bad simulated call costs you those minutes, while a bad real one costs you the carrier.

If you have no scenarios yet, listen to ten real calls and write down what went wrong in each one. Those notes are your first criteria.

## Frequently asked questions

### How do you test a voice AI agent?

Write scenarios from real calls: who calls, what they want and what the agent must and must not do. A simulated caller speaks to your agent over its real transport, with background noise, interruptions and the voice you pick, and a judge passes or fails each criterion from the call transcript and the agent's traces. Run the scenarios on every prompt, model or tool change.

### Can I test a voice agent without writing code?

Yes. In LangWatch you add a voice agent by its ElevenLabs agent id or a phone number, write the scenario in the editor and press Run. Each run shows the verdict per criterion, the recording and the transcript per turn. Voice agents on the platform are in private beta, so on cloud you ask support to turn them on.

### Which voice stacks can Scenario test?

The Scenario SDK has adapters for Pipecat, ElevenLabs, Twilio phone numbers, OpenAI Realtime, Gemini Live and your own speech-to-text, LLM and text-to-speech pipeline. LiveKit, Vapi and raw WebRTC adapters are in preview and Python only. Each one talks to the agent you already deployed, over the protocol it runs on.

### How long does a voice test take, and what does it cost?

A voice scenario runs in real time, so expect about 30 to 120 seconds per run from the SDK and one to two minutes per scenario on the platform. Every call uses your voice provider's minutes, ElevenLabs or Twilio, and a simulated caller also uses OpenAI audio tokens.

### Do I still need to listen to real calls?

Yes, but far fewer. Simulations catch the cases you can describe. Real callers bring cases you never thought of, so trace your production calls, score them with monitors, listen to the ones that fail and turn each one into a new scenario.