Cost per successful task: the metric that will decide your AI stack when model subsidies end

Cost per successful task is the cost of running an AI agent on a task, divided by the rate at which it completes that task correctly. A model that costs $31 per attempt and fully succeeds 3% of the time doesn't cost $31, it costs roughly $1,000 per correct result. It is the only cost metric that matters for agentic workloads, because a cheaper model only saves money if it still produces the outcome you need.

Most teams have never calculated it. Until recently, they didn't have to: flat-rate subscriptions made model choice feel free, so everyone defaulted to whatever scored highest on the leaderboards. That era is ending on a visible timeline, and when it does, this number will dictate your stack. Here's how to calculate it, what the first third-party benchmark data says, and how to use it to move to cheaper models without losing quality.

How to calculate cost per successful task

The formula is simple:

Cost per successful task = (cost per run) ÷ (task success rate)

The inputs are not. You need two things most teams don't systematically track:

• Cost per run - total inference spend for one end-to-end attempt at the task, across every LLM call, tool loop, and retry the agent makes. Not the per-token price, the per-attempt bill.
• Success rate - the fraction of attempts where the outcome actually meets your bar, judged against explicit criteria, not vibes.

A worked example: an agent that costs $0.80 per run and succeeds 78% of the time costs $1.03 per successful task. A "cheaper" alternative at $0.20 per run that succeeds 15% of the time costs $1.33, plus every failure a human has to catch. The cheap model is the expensive one.

What the first benchmark data shows: an 800x spread

In June, Artificial Analysis published AA-Briefcase, the first clean third-party measurement of agentic work costs: 91 private, long-horizon knowledge-work tasks built by practitioners from Google, McKinsey, and BCG. The leaderboard reports cost per task directly:

Model	Cost per task	AA-Briefcase Elo	Cost vs GLM-5.2
Claude Fable 5	~$31	1587	12.9x
Claude Opus 4.8 (max)	$10.40	1356	4.3x
GLM-5.2 (max), open weights	$2.40	1266	1x

Three things stand out. Cost per task spans more than 800x across all models tested, from $31 at the frontier down to $0.04 for a quantized variant that never reaches frontier quality. The leader, Fable 5, satisfies every rubric criterion on just 3% of tasks, putting its cost per fully-correct result near $1,000. And on 31 of the 91 tasks, no model scores above 50%.

The benchmark also shows why frontier pricing is about to become impossible to ignore: these models spend over 100,000 output tokens on a single task. Which brings us to the subsidy problem.

Why this matters now: the subsidy funding your agents is ending

If your agents run on a $200 per month flat plan, someone else is paying most of your inference bill. SemiAnalysis bought every consumer tier from OpenAI and Anthropic and drove each to its weekly limit with agentic workloads: a fully utilized $200 ChatGPT Pro plan consumes up to $14,000 a month in API-equivalent compute, a 70x gap. The flat plan was always a cross-subsidy, with light chat users funding heavy agent users, and it was priced for chat. SemiAnalysis estimates agentic workflows consume up to 1,000x the tokens of a standard query.

Meanwhile, the metered API side is profitable: OpenAI's leaked fiscal 2025 statements show gross margins improving from 28% to 43% (blended, and API-only runs higher), and DeepSeek disclosed a theoretical 545% cost-profit margin on its serving stack. Tokens are cheap to serve. Flat plans lose money because of who uses them and how hard.

And the capital behind the subsidy has a deadline: Epoch AI projects that hyperscaler capex (growing about 70% per year) overtakes their operating cash flow (about 23% per year) around Q3 2026, the point where aggregate free cash flow hits zero. Past that line, the buildout runs on outside capital, and outside capital demands repricing. Anthropic has already introduced metered credit caps for agent usage, and OpenAI, with thinner subscription margins per SemiAnalysis, is the likeliest to move next.

The takeaway isn't panic. It's that the freedom to ignore price and performance tables, the actual product flat plans were selling, is going away. Paying 4.3x for Opus or 12.9x for Fable over GLM-5.2 becomes a per-task decision you have to justify with data.

Benchmarks measure their tasks. You need to measure yours.

AA-Briefcase tells you which model wins on their 91 tasks. Your cost per successful task depends on your workload: your tools, your data, your users, your definition of "done." That number doesn't exist anywhere until you produce it, and producing it takes three layers, the same three layers as the Agent Testing Pyramid:

1. Trace every run. You can't compute cost per run if you can't see the runs. Instrument your agent so every LLM call, tool call, and retry is captured with its token cost. LangWatch tracks cost per trace out of the box, so the numerator is a query, not a spreadsheet.

2. Score every outcome. Define success explicitly, per workflow rather than per prompt, and evaluate against it. LLM-as-judge evals via LangWatch evaluations give you pass or fail at scale on real production traces. That's your denominator.

3. Simulate before you switch. Success rate on historical traces tells you how the current model performs. It can't tell you how a cheaper candidate will perform, and for that you need to run both models through the same multi-turn scenarios. This is what Scenario (open source, Apache 2.0) does: simulated users drive your agent through full conversations, including the failure modes that kill agents in production, the wrong tool call on turn four, the improvised answer when an API times out. Same scenarios, frontier model versus candidate, compared on outcome success rate before any customer sees the downgrade.

Divide spend by pass rate per model and the switching decision writes itself: either the open-weight model holds your success rate at a quarter of the cost, or the frontier premium is justified, per task, with numbers.

What to do before the meter turns on

There's a strange, temporary window right now: frontier baselines are still effectively free to compare against. Use it.

• Instrument everything now. Historical traces are the raw material for every calculation above, and you can't backfill observability.
• Write down what "success" means for each agent workflow, and encode it as evals and simulation pass criteria.
• A/B the downgrade while it's subsidized. Run your simulation suite against the frontier baseline and one or two open-weight candidates (Artificial Analysis flags GLM-5.2 and DeepSeek V4 Pro as the strongest price and performance options) before the bill forces the comparison.
• Keep workloads portable. Model-agnostic agent code plus a test suite that travels turns "move down the curve" into a config change, not a rewrite.

The cheapest your agent will ever be to run is today. Spend the head start on knowing your number, so that when the meter turns on, switching models is an engineering decision backed by data, not a panic.

Want to know your cost per successful task before the meter does? Start with Scenario (open source, Apache 2.0) or book a demo to see the full agent testing and observability stack.