What the improvement loop is
The agent improvement loop is how you change your agent and prove that the change did not make it worse. Change one small thing in the harness, the prompt, a tool description or the model, and something that worked before can break five turns down the line. So you need a loop to keep improving the agent, in cost, in latency or in what it can handle, without dropping the quality. The loop starts with a definition of quality that you can run. That definition is a test suite of scenarios: simulated conversations with the criteria the agent has to meet, all of which pass on the agent you trust today. The judge in each scenario reads the whole conversation together with the agent’s traces, passes or fails every criterion you wrote, and explains the verdict, which is what makes a suite a better gate than a golden answer. Everything else you care about, turns, tokens, cost and latency, is a metric you push down while that suite keeps passing. The same run that gives you the verdicts also gives you those numbers, so you see in one place whether the change paid off and whether it cost you anything.The loop
Repeat from step 3, and put the suite in CI so a later change cannot undo the gain: see Run from CI.
Where the changes come from
The hypotheses come out of the last run. When a criterion failed, the judge’s reasoning quotes the turn where it happened and the trace of that turn shows what the agent saw and did; when a scenario passed but took ten turns, the conversation usually shows the same tool called twice in a row. There are two ways to turn that into a change:- Guided. Langy, or your coding agent with the
langwatchCLI, reads the results and the traces and comes back with a hypothesis and the change for it, and you run the suite on that change to prove it made the agent better. See Prompt optimization with Langy for the prompt case. - Algorithmic. A DSPy optimizer rewrites the instructions and the tool descriptions and scores every candidate by running the suite, and GEPA also feeds the judge’s reasoning back in as the feedback for the next candidate. See Optimize with DSPy.
Guides
Reduce turns
Find out why your agent takes ten turns for a basic task, get the cause changed, and prove the suite still passes.
Fix tool calls
Catch a tool your agent retries, get the tool contract fixed, and assert on the tool calls in the criteria.
Reduce cost and latency
Run the same suite on two models side by side and keep the cheaper one where the pass rate holds.
Optimize with DSPy
Let an optimizer rewrite the instructions and the tool descriptions, scored by your scenarios.
Optimization algorithms
What each algorithm changes, what it needs, what it costs and when to use it.
Prompt optimization with Langy
The same loop on a single prompt, on the experiments workbench, with a dataset and an evaluator.
Also check: Agent testing (how a scenario runs), Connect your agent (one decorator, so the suite runs against your real code), Skills directory (the
agent-improve and scenarios skills for your coding agent).