# Auto-improving agents with LangWatch

Langy, the AI engineer inside LangWatch, can improve your agent automatically for you.

*By Rogerio Chaves · November 13, 2026*

Canonical: https://langwatch.ai/blog/auto-improving-agents

![Auto-improving agents with LangWatch](https://langwatch.ai/blog/auto-improving-agents-cover.webp)

If you build AI agents, you probably run this loop by hand already: read the traces of a bad conversation, guess the cause, change the prompt or the code, rerun the evals, and check for regressions. Langy now runs that loop for you and opens a pull request only when the evals show a win, so you can spend your time on the agent itself.

By now you might have been tempted to set up another agent to improve your agent automatically, but in our experience, agents improving agents unchecked end up with longer prompts, blander answers, and without evals measuring if the changes helped or actually made the agent worse.

**Don't trust AI systems that promise auto self-improvements without hypothesis validation and agent testing. They will always descend into slop.**

So Langy runs each improvement the way a careful engineer would, like an experiment: a hypothesis, an agent test that reproduces the problem, evals against the baseline, and a pull request only when the evals say the change won.

*Figure: one unattended improvement. Langy reads the failing traces and changes the code, pushes the branch, CI runs the evals, the results land in LangWatch and Langy compares them with the baseline. No win: it notes why and tries the next hypothesis. Win: it opens a pull request. When CI finishes on that pull request, LangWatch wakes Langy in the same conversation, up to 3 times in a row.*

## Start from a hypothesis

Langy starts where you would, in the traces of the failing conversations. It writes down what it thinks is wrong, what it will change, and which evals must not regress. It writes this down before changing any code. A bad fix often fixes one case and causes a regression in another. And if you decide what counts as a win after you see the eval results, it's easy to fool yourself.

It also sets up front how many attempts the problem is worth. A small problem shouldn't eat a day of compute.

## Reproduce the failure as a simulation

Langy turns each failing conversation into a simulation: a simulated user replays it and an LLM-as-a-judge grades the result, so there is a test that can prove the hypothesis wrong. Every simulation runs several times. The agent, the simulated user and the judge are all LLMs, so the same simulation can pass once and fail the next time.

If a simulation fails one run in three, something in the agent is unstable. And if a change turns one red run green, that may just be luck.

## Run the evals in your own CI

Langy pushes its branch and lets your CI run the simulations and evals, because only CI runs the real agent, with your keys, data and services. It then reads the eval results back from LangWatch next to the baseline, the same evals run before the change.

*Figure: a GitHub Actions run for Langy's change ticks through checkout, install dependencies, scenario tests and experiment run, then reports to LangWatch: previously failing inputs go from 4/30 passing on the baseline to 24/30, already passing inputs from 30/30 to 29/30. Beside it, the experiment page in LangWatch compares the main run with the langy/rank-titles run, each input run three times: correctness 57% on the baseline, 88% on the branch, held-out inputs stay flat.*

## Langy only opens a PR when the evals prove it's better

Langy also runs the evals on cases it didn't look at, because a fix made for the failing cases sometimes helps those and causes regressions in the others. It opens a pull request only if the change wins on both. If it fails, Langy writes down what it learned and moves to its next hypothesis.

*Figure: a budget of 3 attempts. Attempt 1, keep whole passages: previously failing 4 to 11 of 30 passing, already passing 30 to 27 of 30, no win. Attempt 2, rank titles higher: 4 to 24 of 30, passing 30 to 29 of 30, win, pull request opened. Attempt 3 is not needed.*

## CI wakes Langy up

When CI finishes on its pull request, LangWatch wakes Langy to fix what broke, such as a lint error or a check it didn't run, or to report the result. It gets three of these turns in a row at most, then waits for a person, because an agent answering its own CI could keep chasing a flaky check.

*Figure: on GitHub, lint fails on Langy's pull request. In the Langy panel in LangWatch, CI finishing wakes Langy (follow-up 1 of 3); it fixes the line and pushes, every check passes, follow-up 2 reports the result against the baseline, and the person asks for the next goal.*

## Can you trust the judge?

We test the judge in Scenario, our open-source agent testing library, the same way we test an agent, because every verdict in this loop comes from it, and if the judge is wrong, the whole loop is wrong. The judge looks for evidence in the trace, reads long traces one page at a time, and waits for the agent's last steps to arrive before it decides. It also makes sure every criterion gets a verdict.

Against human labels, a naive LLM-as-a-judge agrees about 87% of the time. The LangWatch Scenario judge harness takes the same models to 94% agreement or higher, which cuts the wrong verdicts by more than half.

| Traces longer than one read | Naive LLM-as-a-judge | LangWatch Scenario judge harness |
| --- | --- | --- |
| gpt-5-mini | 50% | 89% |
| gpt-6-luna | 52% | 100% |
| gpt-6.1-sol | 56% | 100% |
| gemini-3.8-flash | 50% | 100% |

| Evidence in late spans | Naive LLM-as-a-judge | LangWatch Scenario judge harness |
| --- | --- | --- |
| gpt-5-mini | 50% | 100% |
| gpt-6-luna | 50% | 100% |
| gpt-6.1-sol | 50% | 100% |
| gemini-3.8-flash | 50% | 100% |

| All cases | Naive LLM-as-a-judge | LangWatch Scenario judge harness |
| --- | --- | --- |
| gpt-5-mini | 87% | 94% |
| gpt-6-luna | 87% | 98% |
| gpt-6.1-sol | 88% | 98% |
| gemini-3.8-flash | 87% | 98% |

To try it, connect the LangWatch GitHub App and ask Langy to improve a failing scenario. The [Langy pull request docs](https://langwatch.ai/docs/langy/pull-requests) cover the setup.

## Frequently asked questions

### What is an auto-improving agent?

An agent whose code is improved by another agent, without a person in the loop. In LangWatch that is Langy: it forms a hypothesis from the failing traces, reproduces the failure as a test, runs the evals on its change in your CI and opens a pull request only when the evals show it's better. A person still reviews and merges.

### Where does Langy run the evals?

In your repository's own CI. Langy's sandbox can change the code but cannot run your agent, because the keys, data and services it needs live with you. CI has them, so Langy pushes its branch, lets CI run the tests and the evaluation, and reads the results back from LangWatch for the exact commit it pushed.

### What stops Langy from opening pull requests in a loop?

A pull request needs the evals to show a win over the baseline. After CI finishes on a pull request Langy opened, Langy gets at most three automatic turns in a row before it waits for a person, and a daily cap limits how many pull requests it opens.

### Which model should run Langy unattended?

A frontier model, such as Claude Opus 5.5 or gpt-6.1-sol. Reading the traces before forming a hypothesis and running the evals before opening a pull request are the steps that keep the loop on track, and smaller models tend to skip them when no one is watching. The agent being improved keeps its own model.