Auto-improving agents with LangWatch
Langy, the AI engineer inside LangWatch, can improve your agent automatically for you.
Rogerio Chaves · · Agents
If you build AI agents, you probably run this loop by hand already: read the traces of a bad conversation, guess the cause, change the prompt or the code, rerun the evals, and check for regressions. Langy now runs that loop for you and opens a pull request only when the evals show a win, so you can spend your time on the agent itself.
By now you might have been tempted to set up another agent to improve your agent automatically, but in our experience, agents improving agents unchecked end up with longer prompts, blander answers, and without evals measuring if the changes helped or actually made the agent worse.
Don't trust AI systems that promise auto self-improvements without hypothesis validation and agent testing. They will always descend into slop.
So Langy runs each improvement the way a careful engineer would, like an experiment: a hypothesis, an agent test that reproduces the problem, evals against the baseline, and a pull request only when the evals say the change won.
1 of at most 3loop back
Start from a hypothesis
Langy starts where you would, in the traces of the failing conversations. It writes down what it thinks is wrong, what it will change, and which evals must not regress. It writes this down before changing any code. A bad fix often fixes one case and causes a regression in another. And if you decide what counts as a win after you see the eval results, it's easy to fool yourself.
It also sets up front how many attempts the problem is worth. A small problem shouldn't eat a day of compute.
Reproduce the failure as a simulation
Langy turns each failing conversation into a simulation: a simulated user replays it and an LLM-as-a-judge grades the result, so there is a test that can prove the hypothesis wrong. Every simulation runs several times. The agent, the simulated user and the judge are all LLMs, so the same simulation can pass once and fail the next time.
If a simulation fails one run in three, something in the agent is unstable. And if a change turns one red run green, that may just be luck.
Run the evals in your own CI
Langy pushes its branch and lets your CI run the simulations and evals, because only CI runs the real agent, with your keys, data and services. It then reads the eval results back from LangWatch next to the baseline, the same evals run before the change.
Langy only opens a PR when the evals prove it's better
Langy also runs the evals on cases it didn't look at, because a fix made for the failing cases sometimes helps those and causes regressions in the others. It opens a pull request only if the change wins on both. If it fails, Langy writes down what it learned and moves to its next hypothesis.
CI wakes Langy up
When CI finishes on its pull request, LangWatch wakes Langy to fix what broke, such as a lint error or a check it didn't run, or to report the result. It gets three of these turns in a row at most, then waits for a person, because an agent answering its own CI could keep chasing a flaky check.
Ask Langy or describe what you want…
Can you trust the judge?
We test the judge in Scenario, our open-source agent testing library, the same way we test an agent, because every verdict in this loop comes from it, and if the judge is wrong, the whole loop is wrong. The judge looks for evidence in the trace, reads long traces one page at a time, and waits for the agent's last steps to arrive before it decides. It also makes sure every criterion gets a verdict.
Against human labels, a naive LLM-as-a-judge agrees about 87% of the time. The LangWatch Scenario judge harness takes the same models to 94% agreement or higher, which cuts the wrong verdicts by more than half.
To try it, connect the LangWatch GitHub App and ask Langy to improve a failing scenario. The Langy pull request docs cover the setup.
Put this into production with LangWatch.
Trace your agents, run evaluations, and turn failures into repeatable tests.
Frequently asked questions
- What is an auto-improving agent?
- An agent whose code is improved by another agent, without a person in the loop. In LangWatch that is Langy: it forms a hypothesis from the failing traces, reproduces the failure as a test, runs the evals on its change in your CI and opens a pull request only when the evals show it's better. A person still reviews and merges.
- Where does Langy run the evals?
- In your repository's own CI. Langy's sandbox can change the code but cannot run your agent, because the keys, data and services it needs live with you. CI has them, so Langy pushes its branch, lets CI run the tests and the evaluation, and reads the results back from LangWatch for the exact commit it pushed.
- What stops Langy from opening pull requests in a loop?
- A pull request needs the evals to show a win over the baseline. After CI finishes on a pull request Langy opened, Langy gets at most three automatic turns in a row before it waits for a person, and a daily cap limits how many pull requests it opens.
- Which model should run Langy unattended?
- A frontier model, such as Claude Opus 5.5 or gpt-6.1-sol. Reading the traces before forming a hypothesis and running the evals before opening a pull request are the steps that keep the loop on track, and smaller models tend to skip them when no one is watching. The agent being improved keeps its own model.