Refactoring at AI speed, without the AI slop

Why the old app had to change

23 August

LangWatch had grown into a big monolith, with the Vite front end and its server in one package. There were so many different ways and places to put files that you never quite knew where something lived. Prisma was imported in ten different places, and any code could grab the app's shared services from anywhere. So when I changed the Slack reporting code for automations, it broke bug reporting, which is something I'd never have thought to check.

It was slow, too. The old app had 22.1 million type instantiations, and a type check usually took about a minute and twenty seconds, and up to two minutes on a bad run. I could tell when an agent was running one on my laptop, because the audio on my calls would just go to shit.

Agents make that mess worse. If you've worked in a codebase for years, you know which bits the rest of the app depends on. An agent starts every session without that, reads whatever code is nearby and copies it. LLMs also love going off on their own thing. You can give them as many ADRs as you want, and eventually they stop listening. Consistency has always made a codebase easier for people to work in, and it turns out the same goes for LLMs.

The plan was to turn every feature into a small module with its own data, its own tests and a contract (the only part other modules may use), with a machine checking the borders between modules. Then a person or an agent could change one module without reaching into the rest.

After: 66 modules, 53 core and 13 enterprise.

It's basically microservices, just in the same process. The bet was that old-school software engineering, meaning tons of linting, could build a structure the agents aren't allowed to leave. Once the rules are strict enough, the AI gives up arguing and does what it's told, and it keeps going until the lint passes. That let one engineer directing agents do it in weeks while everyone else carried on shipping.

Rules a machine checks

11 September to 12 October

Put ten agents on a codebase at once and you get ten ways of writing the same thing. None of them remember yesterday's conversation, so anything that only lives in someone's head or a review comment gets reinvented the next morning. The rules had to become checks a machine runs, and every agent had to read the same instructions.

By the merge we had 76 custom lint rules, all errors, so there was no pile of warnings to scroll past. They cover file names and locations, imports, route permissions and a five-line limit on comments. Longer explanations belong in an architecture decision record (ADR), with a short link from the code. That keeps long, confident paragraphs out of it. Agents do love writing those.

The architecture enforcer checks what a linter can't see from one file. Its policies look across the whole tree: a module only sees another module's contract, each data-access repository belongs to exactly one module, and no module may end up calling itself through a chain of others. If an agent broke the layout, the build failed.

The errors were prompts too

A rule that only says 'wrong' leaves the agent to invent the fix. Our rule definition gives each lint message three parts: what is wrong, why it matters and what to do instead. The reason has to fit on one line. Here are examples from the rules in the repository:

• Condition shape: if a test makes too many calls, the error counts them and gives the limit. Its reason is that a test you can't read at a glance is where the wrong branch hides. The fix tells the agent to read calls into named constants or split the test into early returns.
• Comment size: a long explanation belongs in an ADR, rather than a block of commentary in the implementation. The error tells the agent where to move it. Shortening the paragraph until it fits isn't the point.
• Application boundaries: if one app imports another app's source, the enforcer names both roles and tells the agent to move the reusable behaviour to its owning feature or infrastructure package. It gives the agent somewhere to put the code, rather than just forbidding the import.

We wrote a lint-rule skill for adding these checks too. It asks for examples that should pass and fail before the implementation, and a fix an agent can apply without guessing. It also sends each check to the right place, so a local syntax check goes in lint, a rule about relationships across packages goes in the architecture enforcer, and taste with no concrete fix stays in the docs instead of becoming another error.

Above both sits the architecture record, one document of over 3,000 lines that says how the code is laid out and why. If the record and the code disagree, the code is wrong. The record also lists the ways of writing things we've retired, so an old pattern can't sneak back in just because an agent found an example of it somewhere.

Human decisions go in the record too, so an agent never has to ask a question that's already been answered. When an agent hits a question the record doesn't answer, it stops and asks, and the answer goes into the record as a numbered decision for later agents to reuse. Feature specs do the same job for behaviour, and if no scenario covers a change, an agent writes one first, error paths included. Skills (the written instructions an agent loads for a kind of task) grew from a handful to over 60.

How we coordinated the work

Every question the agents asked came back to one person, the engineer, around 70 a day through the month from 13 September. They were questions like which module owns this data, whether something should be an event, or whether a difference from main is a bug or an improvement. The agents researched each question first, so a decision usually arrived with the options laid out, what each would cost and which parts of the code it touched.

We put the working instructions in the coordinator skill, the lane agent and the orchestration protocol. The coordinator reads the written work records and gives each lane a bounded assignment: the files it owns, the model it runs on and what done looks like. Lanes work on separate files; shared changes go back through the coordinator. When an agent hits its tool-call limit it writes a handover and stops, and a fresh agent carries on from that record. Questions go to the engineer and the answers go into the architecture record. The work itself lands in small commits.

Agent-hours peaked at about 200 a day on 30 September and on 9 and 10 October, with up to 43 agents running at once.

In mid-September every agent ran its own checks as well as writing code. By 30 September there were 33 agents running at once, which was too much for the laptop, so from then on agents only wrote code and the heavy checks waited in one shared queue. It was also the busiest day for typed messages, with 782 of them.

By the final days, cheap Haiku testers were finding problems and Opus was fixing them, with Codex doing research and second-opinion reviews on the side, all on the same laptop.

Running the whole product locally

Built along the way

The test drive later on meant running the whole product many times over on one laptop. You can't do that if every agent needs a Stripe account, a SAML provider and a credit card, or if forty of them are sharing one database.

A stack for every working copy

haven, our local development tool, starts a separate copy of LangWatch for each working copy of the code: UI, API, worker, databases and stand-ins for outside services. Agents can work side by side without treading on each other, or on the person at the next desk.

• No secrets in sight: agents read the environment through haven, which masks the secrets in it.
• Seeded on start: haven fills an empty stack with users, organisations and logins.
• A feedback button on every page: leave a note, and the agent gets the selector, viewport, console and network calls with it.
• A shared queue for heavy checks, so 40-odd agents don't overload the laptop.

haven itself got a lot lighter, which matters with 40-odd agents on one laptop. Idle CPU fell to about a tenth of what it was, and the shared browser needs about a third of the memory it did.

Stand-ins for outside services

Every outside service LangWatch talks to now has a local stand-in, twelve in all. Tests can make one fail on purpose (a 429 from the LLM, a declined card, a batch of a thousand user-provisioning updates), and we control the responses, which cost nothing. That matters for quality as much as cost, because an agent can actually prove an error path works.

The twelve stand-ins: LLM providers, payments, sign-in (OIDC, SAML and SCIM), passkeys and one-time codes, GitHub, S3 storage, email, Slack and webhooks, analytics, Lambda, voice and telemetry.

One shared headless browser

Agents test the UI through one shared headless Chromium (Playwright's headless shell, GPU off). It serves every agent on the machine, up to 24 at once, each in its own isolated browser session that's already signed in, so an agent never sees a password. For flow testing it reads the page's accessibility tree, which tells it straight away where the Save button is. Screenshots are kept for proof and for visual checks.

We tuned it so two dozen agents can share it on one laptop and still leave room for everything else. Images, fonts, media and animations are off, idle pages and sessions close after a few minutes, and the browser can never take more than an eighth of the machine's RAM.

It can also record any walk through the product as a script, even after it ran (it keeps each session's last 200 actions), and replay it strictly, stopping at the first step that differs. Scripts export to Playwright tests too. We aren't running it yet, but that makes a nightly replay possible. The idea is that an agent walks a flow such as onboarding once, the script replays every night with no AI involved, and an agent only gets involved when it breaks.

Faster checks for the whole team

Measured along the way

Splitting into modules also made checking the code a lot cheaper, and that helps the team building LangWatch every day, the same people who kept shipping through all of this. Before, every type check loaded the whole app as one program. Now each of the 411 TypeScript build projects is checked on its own against its neighbours' already-built types, so a change only rechecks what it touches.

A UI typecheck now takes well under a second, about 30 times faster than midway through the branch, because the compiler does over a hundred times fewer type instantiations (the work of expanding and checking types). Saving a file in development now rebuilds in milliseconds instead of a second or more. A lot of that comes from much smaller generated type files, the ones a package publishes for the packages that use it.

Generated type files: the annotation API went from 911 KB to 3.4 KB, the trace read hook from 930 KB to under 2 KB, and the agent tRPC types from 370 KB to 5 KB.

There's a catch. Checking all 411 projects from cold is about six times the compiler work of the old single app, though you rarely need to. We measured these numbers at individual changes, some of them on a busy machine, so trust the instantiation counts more than the seconds.

Installs got lighter too. Installing the SDK no longer downloads a 38 MB cloudflared binary, and importing the Go SDK brings in 248 packages instead of 389, with no gRPC packages (down from 105).

Testing only what a change affects

CI follows the same borders. It used to run everything on every push. Now Nx works out which projects a change can reach, typecheck keeps every package's built types and only redoes the ones whose source changed, and only the apps a change reaches (API, worker or UI) get tested. If that step fails, all three get tested. Lint only runs on affected projects too, while integration and repository tests are never cached. The rules the agents worked under now run for everyone on the team.

We can't say yet whether CI got faster, because the first app CI runs on main after the merge all failed until a fix landed on 13 October. Before the merge, the median app CI run on main took about 13 minutes, and we'll compare once main is green.