Skip to main content
A refactor, a new lint rule, a CLAUDE.md rewrite or a set of skills all claim to make coding agents work better on your code. You can test that claim: give the same task to the same agent on the codebase before and after the change, and compare what each run cost, how long it took, and what it got wrong. This is a manual experiment. You run both sessions and grade the result yourself; LangWatch records the sessions and gives you the numbers to compare.

​
The run we did

We tested a large layout migration of the LangWatch monorepo (langwatch#7536), which splits the code into feature modules with strict lint rules. The question was whether the new layout made an agent faster, cheaper or more correct. Two Claude Code sessions started 15 seconds apart on 2026-09-08 on the same machine, with the same prompt and models: a Fable 5.1 coordinator that delegated to Opus sub-agents. The task was a full feature: an upload button for image cells in datasets and the evaluation workbench, a file column type, files delivered to prompts and agents as attachments, browser testing with screenshots, then a pull request driven through review. One session ran on the old layout and opened langwatch#7964; the other ran on the new one and opened langwatch#7972. Each run hit one problem unrelated to the layout, and the adjusted rows remove both. The old-layout machine restarted mid-run (about 102 tool calls and 15 USD of repeated work). The new layout’s base branch was missing two files, so the app would not boot until a sub-agent rebuilt them (about 25 minutes and 11 USD).

​
What it showed

Speed: about the same. The new layout finished 5 minutes earlier once the confounds were removed, at the same tool-call rate. Its feedback loop was much faster: scoped typechecks took 5 seconds at the median against 78, so the agent checked twice as often for a third of the waiting time. Cost: about 7% higher on the new layout. It made fewer calls, but each sub-agent carried 29% more context, because one feature now spans several packages and a sub-agent keeps all of them in view. In both runs, the final reports sub-agents write back to the coordinator were about 20% of the cost. Review: the old-layout pull request drew 33 review bot threads across 5 passes before it merged, about 7 per pass. The new-layout pull request drew 11 threads in its one pass and was closed, so the totals measure how long each stayed in review, not which layout drew fewer findings. Defects: we read both final diffs and rated each problem. A severe defect is wrong behaviour on a main path of the feature: data lost or delivered wrong, or a request the server should refuse that it accepts. A minor defect is duplication, misplaced code or an inefficiency that does not change what the user gets. The new layout placed every file where its rules said, and avoided the duplication and one wrong-direction import found in the old run. The new layout still left the three severe defects, all on the run path: a missing attachment fell back silently and the run succeeded, connected agents received a 27 MB base64 string as message text, and the upload endpoint had no request size cap. The old layout left no severe defect; its findings were all minor: a size limit stated in three places, double buffering, a stray results row, and that import. We merged the old-layout pull request. With one run per layout, this is a diagnosis, not proof. Agent behaviour varies a lot between runs of the same prompt. The run did show where to look: the layout steered file placement and loop speed, but neither codebase named the connected-agent path or a size cap as required, so the agent skipped them on both. Those belong in a written spec for the feature.

​
Run it on your codebase

  1. Pick one feature-sized task that touches several parts of the code, and write a single prompt for it that includes testing and opening a pull request.
  2. Prepare two branches: the codebase before and after the change you want to test. Make sure both build and pass CI before you start; a broken base costs the run time and money that tell you about the base, not the change.
  3. Connect Claude Code to LangWatch, then start both sessions at the same time, on the same machine, with the same model settings, each in its own worktree.
  4. Let both open a pull request and go through the same review bot or reviewer.
  5. Compare the two sessions in LangWatch (queries below), and note anything unrelated to the change that cost either run time, so you can subtract it.
  6. Grade the defects. Count review threads, then have one reviewer read both diffs without knowing which is which and rate every defect as severe (wrong behaviour on a main path) or minor. Run the feature yourself.
  7. Repeat with at least three runs per side before you decide. Use the same prompt each time, and start every run in a fresh worktree at the same commit of its branch, with its own pull request branch, so no run starts from an earlier run’s code.

​
Read the numbers in LangWatch

The sessions list shows each session’s cost, duration, model calls and peak context, and the pull requests view shows what each pull request cost. To compare branches side by side, run LangWatchQL with the langwatch command-line tool:
This reads every project your login can read and every session ever recorded on those branches. Pass --project <project> to both commands, and if the branch names were used before, add a time filter: on StartedAt for sessions, and AND TimeUnixMs >= now() - INTERVAL 7 DAY for events. For the feedback loop, compare tool durations per branch:
Your own coding agent can run these too: see Explore your usage with your own agent. Also check: Run your own cost experiments for A/B tests of single settings, and Reduce coding agent cost for the settings worth testing first.
Last modified on October 12, 2026