Things you can ask Langy

Langy is the automated AI engineer inside LangWatch. You ask it about your agent in plain language, and it answers from your real production data and your code. When something is broken, it writes the test, opens the PR, and proves the fix in CI. Here are some of the things people ask it.

"Support says customers are getting refunds they shouldn't. What's happening?"

Langy reads the traces, finds the conversations where it happened, and tells you the cause. On one agent it was the policy check getting skipped whenever retrieval came back empty. Then it offers to fix it. By hand this is an afternoon of grepping traces and reading conversations one at a time.

"Will the cheaper model break anything?"

You want the saving, but a benchmark won't tell you whether the cheaper model still does what your agent needs. Langy turns your real conversations into simulated scenarios and runs them against the candidate model, so you get a per-scenario answer. In one case the cheaper model was 38% cheaper and, in a handful of the 200 scenarios, it stopped asking for the order number before issuing a refund. You catch it in testing instead of in production.

"Why did our LLM bill triple last month?"

Langy traces it to the line: on one team's agent, a single step was re-sending the whole order history every turn, 71% of the spend, on context the model had already seen. It cached it and the bill came down about 60%. You get the cause and the change that fixes it.

"Did my agent get worse this week?"

This is the hard one to answer by hand, because "worse" is spread across thousands of conversations. Langy reads them, finds where users got misunderstood or gave up, and points at the change that moved the number. Then it writes a test that catches the regression, so it can't come back quietly.

You can ask narrower things too: which prompts regressed after a library bump, whether the Portuguese conversations do worse than the English ones, if anything is writing a phone number into a log. Same shape every time: you ask, it goes through the real data, it comes back with something you can act on.

Why it can actually do this

First, Langy is wired into everything it needs. It lives on your traces and your evals, and it connects to your GitHub. It reads the same production data you read and opens PRs against the same code you ship, so it reasons about your agent the way someone who has been on the team a while does, not a chatbot working from a snippet you pasted.

Second, the harness. Langy carries the three years we have spent getting hundreds of teams' agents into production: how to write a simulation test, what a trustworthy eval looks like, how to turn a bad trace into a failing test. That is the actual job, and it is built into how Langy works, not something it rediscovers on your repo each time.

"Could I do this with my own Claude Code?"

Yes, absolutely. That's the point. We give you the LangWatch CLI so your own Claude can reach everything on the platform, and Scenario, our open-source framework, for the simulation testing. Wire those together and your Claude becomes a genuinely good AI engineer for this, and honestly, we want you to. Langy is the same thing with the harness already assembled and sitting on your data, so you can skip the wiring and just use it. And soon you will be able to point your own Claude straight at Langy and run it as a subagent, instead of calling the CLIs and MCPs yourself.

You don't have to be an engineer to ask

Because Langy lives on the platform, the person asking does not have to write code. A PM who knows the product, or a support lead who knows exactly how it fails, can ask and get an engineered change back with the tests behind it. That is more agency for the people who know the domain best, and it leaves the engineers the harder problems instead of the triage.