Frontier LLMs are up to 22 points more accurate than Jev, but cost over 100x more

Every step up the Pareto frontier buys accuracy at a much higher price. Run a cheap judge on every trace, and save the frontier models for the cases where the extra points matter.

This benchmark was created using LangWatch. Sign up to create your own · Read the method

LangWatch Instant Evals runs on Jev. Check it out

Task
Set

LLM-as-a-judge, core

Accuracy, % · Higher is better

Pick: GPT-6.1 Sol, tied with Claude Opus 5.5 at about half its cost.

  1. , interval 89.1 to 95.8, tied for the lead
  2. , interval 83.9 to 93.9, tied for the lead
  3. , interval 83.2 to 90.6, tier 2
  4. , interval 79.7 to 89.1, tier 2
  5. , interval 78.4 to 88.5, tier 2
  6. , interval 77.5 to 87.8, tier 2
  7. , interval 76.2 to 85.5, tier 3
  8. , interval 67.9 to 80.3, tier 3

Ahead aloneTied for the leadThe rest95% interval

One job at a time

An evaluator applies its criterion to a trace or a thread: pass or fail. A balanced sample of labelled cases from six demo agents, half true and half false.

LLM-as-a-judge, core: Accuracy, higher is better

  1. GPT-6.1 Sol92.6%, interval 92.6% [89.1, 95.8], tier 1
  2. Claude Opus 5.589.1%, interval 89.1% [83.9, 93.9], tier 1
  3. Gemini 4 Hydrogen87.0%, interval 87.0% [83.2, 90.6], tier 2
  4. Gemini 4 Argon84.4%, interval 84.4% [79.7, 89.1], tier 2
  5. Claude Sonnet 5.583.5%, interval 83.5% [78.4, 88.5], tier 2
  6. GPT-6 Luna82.6%, interval 82.6% [77.5, 87.8], tier 2
  7. Gemini 3.8 Flash80.9%, interval 80.9% [76.2, 85.5], tier 3
  8. Claude Haiku 5.573.9%, interval 73.9% [67.9, 80.3], tier 3
  9. OpenAI Decisions API71.7%, interval 71.7% [66.1, 77.2], tier 4
  10. Jev62.0%, interval 62.0% [56.1, 68.3], tier 5
Dot: accuracy. Line: its 95% interval. Higher is better. Orange: ahead by the tie test; green: tied for the lead. Axis starts at 50%, not 0.
Plot

LLM-as-a-judge, core: accuracy (higher is better) vs cost

↖ Better
  • Jev
  • GPT-6.1 Sol
  • GPT-6 Luna
  • OpenAI Decisions API
  • Gemini 4 Hydrogen
← Cost per 1,000 decisions, USD (log scale). Lower is betterHigher is better ↑

Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~).

Every model, every job

Every visible model on every task, sorted by LLM-as-a-judge, core. Value with half its interval; bands are tie tiers.
Model
Tier 1 on LLM-as-a-judge, core
GPT-6.1 Sol87.8±2.8 · T187.8% [84.8, 90.4], tier 192.7±2.8 · T292.7% [89.7, 95.4], tier 296.8±1.5 · T296.8% [95.2, 98.2], tier 292.6±3.4 · T192.6% [89.1, 95.8], tier 196.9±3.5 · T196.9% [93.0, 100.0], tier 198.5±1.7 · T198.5% [96.6, 100.0], tier 181.3±6.6 · T281.3% [74.1, 87.3], tier 286.5±6.4 · T286.5% [79.8, 92.5], tier 296.0±3.3 · T196.0% [92.4, 99.0], tier 189.6±3.8 · T189.6% [85.6, 93.1], tier 194.8±4.5 · T194.8% [90.0, 98.9], tier 196.0±2.6 · T296.0% [93.3, 98.5], tier 2
Claude Opus 5.589.2±2.9 · T189.2% [86.0, 91.8], tier 189.2±3.8 · T389.2% [85.2, 92.8], tier 397.8±1.2 · T297.8% [96.5, 98.8], tier 289.1±5.0 · T189.1% [83.9, 93.9], tier 192.7±6.0 · T192.7% [85.9, 97.9], tier 199.0±1.5 · T199.0% [97.0, 100.0], tier 186.8±6.1 · T186.8% [80.2, 92.3], tier 182.3±8.2 · T282.3% [74.0, 90.4], tier 297.0±2.2 · T197.0% [94.6, 99.0], tier 191.5±3.6 · T191.5% [87.6, 94.8], tier 192.7±4.8 · T192.7% [87.5, 97.1], tier 197.5±2.2 · T197.5% [95.0, 99.5], tier 1
Tier 2
Gemini 4 Hydrogen87.7±2.5 · T187.7% [85.0, 90.1], tier 189.2±3.5 · T389.2% [85.4, 92.4], tier 399.0±0.7 · T199.0% [98.1, 99.6], tier 187.0±3.7 · T287.0% [83.2, 90.6], tier 293.8±4.8 · T193.8% [88.3, 98.0], tier 1100.0±0.0 · T1100.0% [100.0, 100.0], tier 186.1±5.9 · T186.1% [79.8, 91.6], tier 182.3±7.8 · T282.3% [74.0, 89.6], tier 298.0±1.8 · T198.0% [95.9, 99.5], tier 190.0±2.9 · T190.0% [86.9, 92.8], tier 191.7±5.1 · T191.7% [85.9, 96.2], tier 199.0±1.3 · T199.0% [97.5, 100.0], tier 1
Gemini 4 Argon83.4±2.8 · T283.4% [80.4, 86.1], tier 295.1±2.4 · T195.1% [92.6, 97.4], tier 198.2±1.0 · T198.2% [97.0, 99.0], tier 184.4±4.7 · T284.4% [79.7, 89.1], tier 295.8±3.8 · T195.8% [91.5, 99.0], tier 199.0±1.3 · T199.0% [97.4, 100.0], tier 184.0±5.9 · T184.0% [77.7, 89.6], tier 194.8±4.3 · T194.8% [90.4, 98.9], tier 196.5±2.3 · T196.5% [93.9, 98.6], tier 181.7±3.8 · T281.7% [77.9, 85.4], tier 294.8±4.4 · T194.8% [90.2, 98.9], tier 199.0±1.3 · T199.0% [97.5, 100.0], tier 1
Claude Sonnet 5.584.6±3.0 · T284.6% [81.5, 87.5], tier 288.9±3.1 · T388.9% [85.5, 91.8], tier 395.0±1.7 · T395.0% [93.2, 96.6], tier 383.5±5.0 · T283.5% [78.4, 88.5], tier 295.8±3.8 · T195.8% [91.3, 99.0], tier 199.0±1.3 · T199.0% [97.5, 100.0], tier 181.3±6.7 · T281.3% [74.1, 87.4], tier 277.1±7.1 · T377.1% [69.8, 84.0], tier 389.5±4.1 · T289.5% [85.2, 93.4], tier 289.1±3.2 · T189.1% [85.8, 92.1], tier 193.8±4.8 · T193.8% [88.3, 97.9], tier 196.5±2.5 · T196.5% [93.9, 99.0], tier 1
GPT-6 Luna79.2±3.2 · T379.2% [76.0, 82.3], tier 386.8±3.2 · T386.8% [83.5, 89.8], tier 395.0±1.7 · T395.0% [93.3, 96.6], tier 382.6±5.1 · T282.6% [77.5, 87.8], tier 293.8±4.7 · T193.8% [88.5, 97.9], tier 197.5±2.0 · T297.5% [95.5, 99.5], tier 279.9±6.7 · T279.9% [72.8, 86.2], tier 272.9±6.9 · T372.9% [66.0, 79.8], tier 393.5±3.3 · T293.5% [90.1, 96.7], tier 275.2±4.4 · T375.2% [70.7, 79.6], tier 393.8±4.6 · T193.8% [88.8, 97.9], tier 194.0±3.3 · T294.0% [90.5, 97.1], tier 2
Tier 3
Gemini 3.8 Flash83.1±2.8 · T283.1% [80.1, 85.8], tier 294.8±2.3 · T194.8% [92.2, 96.8], tier 197.3±1.2 · T297.3% [96.0, 98.4], tier 280.9±4.6 · T380.9% [76.2, 85.5], tier 395.8±3.7 · T195.8% [91.5, 99.0], tier 199.5±0.8 · T199.5% [98.4, 100.0], tier 183.3±6.2 · T183.3% [76.8, 89.1], tier 195.8±3.6 · T195.8% [91.8, 99.0], tier 195.5±2.6 · T195.5% [92.8, 97.9], tier 185.0±3.5 · T285.0% [81.3, 88.4], tier 292.7±4.7 · T192.7% [87.5, 97.0], tier 197.0±2.3 · T197.0% [94.4, 99.1], tier 1
Claude Haiku 5.577.0±3.6 · T377.0% [73.4, 80.5], tier 383.0±3.4 · T483.0% [79.5, 86.3], tier 489.0±2.4 · T489.0% [86.6, 91.3], tier 473.9±6.2 · T373.9% [67.9, 80.3], tier 394.8±4.6 · T194.8% [89.8, 98.9], tier 198.5±1.7 · T198.5% [96.6, 100.0], tier 175.7±7.7 · T275.7% [67.8, 83.2], tier 266.7±6.5 · T466.7% [60.4, 73.4], tier 473.5±6.1 · T373.5% [67.2, 79.5], tier 381.3±4.2 · T281.3% [76.9, 85.4], tier 287.5±6.4 · T287.5% [80.8, 93.6], tier 295.0±3.1 · T295.0% [91.8, 98.0], tier 2
Tier 4
OpenAI Decisions API75.2±3.3 · T475.2% [71.8, 78.4], tier 470.8±4.5 · T570.8% [66.3, 75.3], tier 575.0±3.4 · T575.0% [71.5, 78.3], tier 571.7±5.6 · T471.7% [66.1, 77.2], tier 471.9±8.7 · T271.9% [62.8, 80.2], tier 277.0±5.5 · T377.0% [71.3, 82.2], tier 383.3±6.9 · T183.3% [76.1, 89.9], tier 162.5±7.6 · T462.5% [55.1, 70.2], tier 473.5±6.4 · T373.5% [67.0, 79.9], tier 370.4±4.4 · T370.4% [66.0, 74.8], tier 378.1±6.9 · T378.1% [71.3, 85.1], tier 374.5±5.9 · T374.5% [68.4, 80.2], tier 3
Tier 5
Jev66.7±3.9 · T566.7% [62.8, 70.5], tier 573.6±4.7 · T573.6% [68.9, 78.2], tier 574.8±3.4 · T574.8% [71.3, 78.1], tier 562.0±6.1 · T562.0% [56.1, 68.3], tier 572.9±8.1 · T272.9% [64.5, 80.8], tier 276.7±5.9 · T376.7% [70.6, 82.5], tier 369.9±8.5 · T369.9% [61.0, 78.0], tier 374.3±7.5 · T374.3% [66.7, 81.7], tier 374.7±5.8 · T374.7% [68.7, 80.3], tier 368.2±5.0 · T468.2% [63.0, 73.1], tier 473.6±8.5 · T373.6% [65.0, 82.0], tier 373.0±6.1 · T373.0% [66.7, 79.0], tier 3

Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. No overall column: tasks use different metrics, so we don't average them.

How we measured

Scores carry a 95% bootstrap interval (2,000 resamples, the versions of one trace or scenario resampled together). Two models are tied when the 95% interval of their paired difference on the same cases holds zero.

Tie tiers

Models in one tier can't be separated by the tie test (the 95% interval of the paired difference on the same cases holds zero). Tiers are computed in the export over every model; filters on this page never change them.

Contamination

The cases are labelled traces from LangWatch’s own demo agents. They were never published, so no model can have trained on them. The page shows aggregates only: no case and no case text.

Latency and hardware

  • Hosted API: Jev, Gemini 3.8 Flash, GPT-6 Luna, OpenAI Decisions API, Claude Haiku 5.5, Gemini 4 Argon and Gemini 4 Hydrogen, one API call per judgment, several at a time
  • Vendor CLI: GPT-6.1 Sol, Claude Opus 5.5 and Claude Sonnet 5.5, through their vendors' command-line tools; each call's time includes the tool's start-up

Latencies are only compared within one hardware tier.

Datasets

  • All three jobs, core (834 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • All three jobs, long conversations (288 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • All three jobs, hard cases (600 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • LLM-as-a-judge, core (230 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • LLM-as-a-judge, long conversations (96 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • LLM-as-a-judge, hard cases (200 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Scenario judge, core (144 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Scenario judge, long conversations (96 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Scenario judge, hard cases (200 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Search, core (460 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Search, long conversations (96 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • Search, hard cases (200 test items): labelled traces from LangWatch demo agents (private, aggregates only)
  • A refusal, or a reply with no verdict in it, counts as wrong, as it would for a customer.
  • Cost and latency were measured on the core set only, one call per judgment.
  • All three jobs: the mean of the three job scores on the set, each job counting the same; its interval and the tie test combine the jobs’ intervals, and its cost is the mean of the three job costs.
  • Jev, Gemini 3.8 Flash, GPT-6 Luna, OpenAI Decisions API, Claude Haiku 5.5, Gemini 4 Argon and Gemini 4 Hydrogen: cost recorded on the run. GPT-6.1 Sol, Claude Opus 5.5 and Claude Sonnet 5.5 ran on subscriptions, so their cost is an estimate (~) at the API list price; priced on the tokens the CLI reported, it would be up to $188 per 1,000 judgments.
  • GPT-6.1 Sol, Claude Opus 5.5 and Claude Sonnet 5.5 ran through their vendors' command-line tools, so each call's time includes the tool's start-up. Their latency is shown apart from the API calls (Jev, Gemini 3.8 Flash, GPT-6 Luna, OpenAI Decisions API, Claude Haiku 5.5, Gemini 4 Argon and Gemini 4 Hydrogen) and never compared with them.
  • Jev: Results are for Jev 1.14.0, called on 15 October 2026, three passes over every check; its score is the mean of the passes. The passes sent more calls at once than the earlier runs, so its times are not from the same setup as the other models’. Every other model’s answers are as before.
  • GPT-6.1 Sol: Run through the Codex CLI on a ChatGPT subscription, reasoning effort low. Codex adds its own system prompt in front of ours, and the forced function call becomes a JSON reply. Latency includes the CLI start‑up; cost is an estimate at the API list price.
  • Claude Opus 5.5: Run through Claude Code on a Claude subscription. The forced function call becomes a JSON reply. Latency includes the CLI start‑up; cost is an estimate at the API list price. A refusal by the CLI’s safeguards counts as wrong; the API may refuse less often.
  • Claude Sonnet 5.5: Run through Claude Code on a Claude subscription. The forced function call becomes a JSON reply. Latency includes the CLI start‑up; cost is an estimate at the API list price. A refusal by the CLI’s safeguards counts as wrong; the API may refuse less often.
  • OpenAI Decisions API: Called through OpenAI’s Decisions API, in public beta, which runs GPT-6 Luna. It gets the typed question Jev gets, over the same text Jev reads, and answers with a probability for each answer instead of text. Billed on input tokens only.
  • Gemini 3.8 Flash: Metered at the introductory price, which runs until 31 December 2026; the list price roughly doubles after that.
  • Gemini 4 Argon: Metered at the introductory price, which runs until 30 November 2026; the list price doubles after that.
  • We also re-ran the other judges with the richest input that fits their context window; no LLM moved by more than 8.3 points, but a pick changed. Claude Haiku 5.5, Gemini 4 Argon and Gemini 4 Hydrogen ran the production input only.

Release production-2026-11-14, integrity sha256-leaves-v1.