OpenAI Decisions API beats Jev on real-world checks

LangWatch LLM-as-a-Judge Bench evaluates model capabilities in doing evals, agent simulations and trace search, our specialty. Core and long conversations are real-world based, hard cases synthetic; none is public, to avoid benchmaxing and measure true capability for our customers.

This benchmark was created using LangWatch. Sign up to create your own · Read the method

LangWatch Instant Evals runs on Jev. Check it out

Task
Set

All three jobs, core

Accuracy, % · Higher is better

OpenAI Decisions API is clearly ahead of Jev here, by about 6 points.

  1. , interval 69.3 to 76.6, not ranked
  2. , interval 70.0 to 76.2, ahead alone
  3. , interval 64.7 to 71.8, tier 2
  4. Jev66.8
    , interval 63.2 to 70.2, tier 2
  5. GLiDE65.2
    , interval 62.0 to 68.5, tier 2
  6. , interval 59.9 to 66.3, tier 3
  7. , interval 59.4 to 66.5, tier 3
  8. , interval 59.1 to 66.1, tier 3

Ahead aloneTied for the leadThe rest95% interval

  • On real conversations OpenAI Decisions API and Mercury Decide score highest, for different reasons. The OpenAI Decisions API misses fewer real problems, above all facts the agent made up and mistakes made earlier in the conversation; Mercury Decide less often flags a response that is fine.
  • When an agent says it did something it never did, such as opening a ticket, no decision model reliably notices: the best catches only about two in five of these, and an LLM does little better.

What each model catches

The reply states a price, a setting or a step that no tool or document returned.

The share of checks each model gets right; deeper orange is a higher share. Models are in the order of the chart above. A long input is compacted to fit a model’s window, the same way for every model; a model not yet rerun that way counts only the checks it read in full, and is not listed without enough of them for every column.

One job at a time

All three jobs, core: Accuracy, higher is better

  1. Vega 27B Fine-Tuned (LangWatch)73.0%, interval 73.0% [69.3, 76.6], not ranked
  2. OpenAI Decisions API73.0%, interval 73.0% [70.0, 76.2], tier 1
  3. Mercury Decide68.2%, interval 68.2% [64.7, 71.8], tier 2
  4. Jev66.8%, interval 66.8% [63.2, 70.2], tier 2
  5. GLiDE65.2%, interval 65.2% [62.0, 68.5], tier 2
  6. deck-31B63.1%, interval 63.1% [59.9, 66.3], tier 3
  7. Kev-27B62.8%, interval 62.8% [59.4, 66.5], tier 3
  8. pplx-decider v1.1 27B62.6%, interval 62.6% [59.1, 66.1], tier 3
  9. Clef62.4%, interval 62.4% [59.0, 65.6], tier 3
  10. Decision 2.0 Vega 27B61.0%, interval 61.0% [57.4, 64.9], tier 3
  11. Gemini Decision 159.4%, interval 59.4% [55.3, 63.4], tier 3
  12. Microsoft Decision 158.5%, interval 58.5% [54.6, 62.6], tier 3
  13. Winnow58.5%, interval 58.5% [54.9, 62.2], tier 4
  14. Quyet-1.0-Large58.4%, interval 58.4% [54.9, 62.1], tier 4
  15. Rene-1 31B FP858.4%, interval 58.4% [54.4, 62.3], tier 4
  16. H2O-Lightning-4B v1.2.257.4%, interval 57.4% [53.3, 61.5], tier 4
  17. AutoJev-27B55.6%, interval 55.6% [51.9, 59.4], tier 4
  18. Jev-Omni53.6%, interval 53.6% [49.9, 57.4], tier 5
  19. decisio v0.8.051.9%, interval 51.9% [48.1, 56.0], tier 5
  20. Decision 2.0 Nox 4B51.2%, interval 51.2% [47.3, 55.5], tier 5
  21. Cygnet51.1%, interval 51.1% [47.2, 55.2], tier 5
  22. decider-4b v250.0%, interval 50.0% [45.8, 54.3], tier 5
  23. JevK549.8%, interval 49.8% [46.0, 53.7], tier 5
  24. DiffusionGemma-Jev48.7%, interval 48.7% [44.5, 53.0], tier 5
  25. Clef Omni48.4%, interval 48.4% [44.6, 52.5], tier 6
  26. Hopper 12B48.3%, interval 48.3% [44.5, 52.5], tier 6
  27. Julia-147.4%, interval 47.4% [43.7, 50.9], tier 6
  28. Clef Flash47.1%, interval 47.1% [42.8, 51.6], tier 6
  29. Decision 2.0 Lux 9B45.4%, interval 45.4% [41.6, 49.7], tier 6
  30. Decision 2.0 Sol 2B45.1%, interval 45.1% [40.8, 49.5], tier 6
  31. Decision 2.0 Eos 0.8B44.8%, interval 44.8% [41.0, 49.0], tier 6
  32. Decision 2.0 Kai 0.6B43.5%, interval 43.5% [39.3, 48.0], tier 6
  33. Lev 4B42.1%, interval 42.1% [38.1, 46.2], tier 7
Dot: accuracy. Line: its 95% interval. Higher is better. Orange: ahead by the tie test; green: tied for the lead. Axis starts at 30%, not 0.
Plot

All three jobs, core: accuracy (higher is better) vs cost

Orange: most accurate decision model; blue: LLMs for reference

↖ Better
  • Jev
  • OpenAI Decisions API
  • Opus
  • Sol
  • Luna
  • Flash
  • Argon
  • Hydrogen
← Cost per 1,000 decisions, USD (log scale). Lower is betterHigher is better ↑

Blue LLMs are not ranked. Opus and Sol (hollow): an estimate at API list price. LLM cost is one figure over all three sets.

Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).

Jev alternatives, every job

Every visible model on every task, sorted by All three jobs, core. Value with half its interval; bands are tie tiers.
Model
Not ranked on this task
Vega 27B Fine-Tuned (LangWatch)73.0±3.673.0% [69.3, 76.6], not ranked67.0±7.367.0% [59.9, 74.5], not ranked75.6±4.775.6% [70.8, 80.3], not ranked74.3±8.374.3% [65.4, 82.0], not ranked79.2±4.379.2% [74.7, 83.3], not ranked77.1±7.577.1% [69.0, 84.0], not ranked82.3±7.182.3% [75.0, 89.1], not ranked78.1±7.978.1% [70.2, 86.0], not ranked81.0±3.181.0% [78.1, 84.3], not ranked80.5±5.680.5% [74.9, 86.0], not ranked80.0±5.080.0% [75.0, 85.0], not ranked82.5±5.182.5% [77.3, 87.5], not ranked77.7±2.277.7% [75.5, 79.9], not ranked74.9±4.174.9% [70.6, 78.8], not ranked79.3±3.379.3% [76.0, 82.6], not ranked78.3±4.078.3% [74.1, 82.2], not ranked
Tier 1 on All three jobs, core
OpenAI Decisions API73.0±3.1 · T173.0% [70.0, 76.2], tier 171.7±5.6 · T171.7% [66.1, 77.2], tier 170.4±4.4 · T170.4% [66.0, 74.8], tier 183.3±6.9 · T183.3% [76.1, 89.9], tier 170.8±4.6 · T270.8% [66.2, 75.3], tier 271.9±8.7 · T171.9% [62.8, 80.2], tier 178.1±6.9 · T178.1% [71.3, 85.1], tier 162.5±7.6 · T262.5% [55.1, 70.2], tier 275.0±3.4 · T275.0% [71.7, 78.5], tier 277.0±5.5 · T177.0% [71.3, 82.2], tier 174.5±5.9 · T274.5% [68.4, 80.2], tier 273.5±6.4 · T373.5% [67.0, 79.9], tier 373.0±2.2 · T273.0% [70.7, 75.1], tier 273.5±3.9 · T173.5% [69.6, 77.4], tier 174.4±3.5 · T174.4% [70.8, 77.7], tier 173.1±4.0 · T273.1% [69.2, 77.1], tier 2
Tier 2
Mercury Decide68.2±3.6 · T268.2% [64.7, 71.8], tier 262.6±6.2 · T262.6% [56.6, 69.1], tier 266.3±4.7 · T166.3% [61.5, 71.0], tier 183.3±7.4 · T183.3% [75.5, 90.3], tier 178.8±4.7 · T178.8% [74.1, 83.6], tier 181.3±8.0 · T181.3% [72.9, 89.0], tier 180.2±8.1 · T180.2% [71.7, 88.0], tier 175.0±8.2 · T175.0% [66.7, 83.0], tier 178.7±3.4 · T278.7% [75.3, 82.1], tier 279.0±6.5 · T179.0% [72.2, 85.2], tier 176.0±6.0 · T176.0% [69.8, 81.8], tier 181.0±5.4 · T281.0% [75.7, 86.5], tier 275.2±2.3 · T175.2% [72.8, 77.4], tier 174.3±4.0 · T174.3% [70.3, 78.4], tier 174.2±3.7 · T174.2% [70.3, 77.7], tier 179.8±4.0 · T179.8% [75.6, 83.6], tier 1
Jev66.8±3.5 · T266.8% [63.2, 70.2], tier 262.0±6.1 · T262.0% [56.1, 68.3], tier 268.2±5.0 · T168.2% [63.0, 73.1], tier 169.9±8.5 · T269.9% [61.0, 78.0], tier 273.6±4.8 · T173.6% [68.8, 78.3], tier 172.9±8.1 · T172.9% [64.5, 80.8], tier 173.6±8.5 · T273.6% [65.0, 82.0], tier 274.3±7.5 · T174.3% [66.7, 81.7], tier 174.8±3.5 · T374.8% [71.3, 78.2], tier 376.7±5.9 · T276.7% [70.6, 82.5], tier 273.0±6.1 · T273.0% [66.7, 79.0], tier 274.7±5.8 · T374.7% [68.7, 80.3], tier 371.7±2.3 · T271.7% [69.3, 74.0], tier 270.5±3.9 · T170.5% [66.5, 74.4], tier 171.6±3.9 · T171.6% [67.6, 75.5], tier 173.0±4.4 · T273.0% [68.5, 77.3], tier 2
GLiDE65.2±3.2 · T265.2% [62.0, 68.5], tier 259.1±6.4 · T259.1% [52.9, 65.6], tier 265.2±4.3 · T165.2% [60.9, 69.4], tier 175.0±8.1 · T275.0% [66.7, 82.9], tier 279.5±4.5 · T179.5% [75.0, 83.9], tier 178.1±7.7 · T178.1% [70.4, 85.9], tier 185.4±6.3 · T185.4% [79.0, 91.7], tier 175.0±8.2 · T175.0% [66.7, 83.0], tier 174.3±3.5 · T374.3% [70.9, 78.0], tier 372.0±6.2 · T272.0% [65.7, 78.1], tier 271.5±6.7 · T271.5% [64.4, 77.8], tier 279.5±5.1 · T279.5% [74.4, 84.7], tier 273.0±2.2 · T273.0% [70.8, 75.2], tier 269.8±3.9 · T169.8% [65.8, 73.6], tier 174.0±3.3 · T174.0% [70.7, 77.3], tier 176.5±4.0 · T276.5% [72.3, 80.4], tier 2
Tier 3
deck-31B63.1±3.2 · T363.1% [59.9, 66.3], tier 359.6±5.6 · T259.6% [54.3, 65.6], tier 258.9±4.6 · T258.9% [54.3, 63.5], tier 281.9±6.7 · T181.9% [74.8, 88.1], tier 168.8±5.0 · T268.8% [63.8, 73.8], tier 268.8±7.9 · T168.8% [60.8, 76.6], tier 161.5±9.4 · T261.5% [52.0, 70.8], tier 276.0±7.6 · T176.0% [68.1, 83.3], tier 181.5±3.2 · T181.5% [78.4, 84.8], tier 180.0±6.4 · T180.0% [73.4, 86.2], tier 177.5±6.3 · T177.5% [71.0, 83.7], tier 187.0±4.5 · T187.0% [82.5, 91.5], tier 171.1±2.2 · T271.1% [68.8, 73.3], tier 269.4±3.9 · T169.4% [65.5, 73.4], tier 166.0±4.1 · T266.0% [61.8, 70.1], tier 281.7±3.7 · T181.7% [77.8, 85.2], tier 1
Kev-27B62.8±3.6 · T362.8% [59.4, 66.5], tier 359.1±7.0 · T259.1% [52.4, 66.3], tier 261.1±4.7 · T261.1% [56.4, 65.7], tier 274.3±7.8 · T274.3% [66.4, 82.0], tier 275.3±4.7 · T175.3% [70.8, 80.1], tier 171.9±7.8 · T171.9% [64.3, 79.8], tier 177.1±8.5 · T177.1% [68.4, 85.4], tier 177.1±7.7 · T177.1% [69.0, 84.4], tier 172.0±3.4 · T372.0% [68.5, 75.4], tier 372.0±6.4 · T272.0% [65.6, 78.4], tier 266.0±6.0 · T266.0% [59.7, 71.8], tier 278.0±6.2 · T278.0% [71.6, 84.0], tier 270.1±2.4 · T370.1% [67.8, 72.5], tier 367.7±4.0 · T267.7% [63.8, 71.8], tier 268.1±3.7 · T268.1% [64.3, 71.8], tier 276.5±4.2 · T276.5% [72.2, 80.5], tier 2
pplx-decider v1.1 27B62.6±3.5 · T362.6% [59.1, 66.1], tier 361.3±5.5 · T261.3% [56.0, 67.0], tier 258.9±4.8 · T258.9% [54.1, 63.7], tier 276.4±7.6 · T176.4% [68.5, 83.7], tier 170.8±4.8 · T270.8% [66.1, 75.7], tier 272.9±8.0 · T172.9% [64.9, 80.8], tier 171.9±8.6 · T271.9% [63.3, 80.4], tier 267.7±7.7 · T167.7% [60.2, 75.5], tier 183.7±2.9 · T183.7% [80.8, 86.6], tier 184.0±5.0 · T184.0% [79.0, 88.9], tier 182.5±5.5 · T182.5% [77.0, 87.9], tier 184.5±4.9 · T184.5% [79.6, 89.4], tier 172.4±2.2 · T272.4% [70.2, 74.5], tier 272.7±3.7 · T172.7% [68.9, 76.3], tier 171.1±3.8 · T171.1% [67.3, 74.9], tier 176.2±4.1 · T276.2% [72.1, 80.2], tier 2
Clef62.4±3.3 · T362.4% [59.0, 65.6], tier 358.3±6.0 · T258.3% [52.3, 64.3], tier 260.2±4.6 · T260.2% [55.7, 64.9], tier 275.7±7.9 · T175.7% [67.4, 83.1], tier 176.7±4.5 · T176.7% [72.3, 81.3], tier 176.0±7.5 · T176.0% [68.4, 83.3], tier 183.3±7.0 · T183.3% [76.0, 90.0], tier 170.8±8.5 · T170.8% [62.2, 79.2], tier 174.0±3.5 · T374.0% [70.5, 77.5], tier 376.5±6.5 · T276.5% [69.9, 83.0], tier 266.5±5.7 · T266.5% [60.8, 72.2], tier 279.0±5.3 · T279.0% [73.6, 84.3], tier 271.0±2.3 · T271.0% [68.7, 73.3], tier 270.3±3.8 · T170.3% [66.8, 74.3], tier 170.0±3.3 · T170.0% [66.6, 73.3], tier 175.2±4.3 · T275.2% [70.7, 79.3], tier 2
Decision 2.0 Vega 27B61.0±3.7 · T361.0% [57.4, 64.9], tier 360.4±6.3 · T260.4% [54.3, 67.0], tier 258.3±5.3 · T258.3% [53.1, 63.7], tier 270.8±8.1 · T270.8% [62.4, 78.7], tier 280.2±4.3 · T180.2% [76.0, 84.6], tier 176.0±8.2 · T176.0% [67.7, 84.0], tier 184.4±6.4 · T184.4% [77.8, 90.6], tier 180.2±6.7 · T180.2% [73.4, 86.7], tier 175.8±3.5 · T275.8% [72.5, 79.4], tier 275.5±5.7 · T275.5% [69.8, 81.3], tier 271.0±6.2 · T271.0% [64.5, 77.0], tier 281.0±5.3 · T281.0% [75.5, 86.1], tier 272.4±2.2 · T272.4% [70.1, 74.5], tier 270.7±4.0 · T170.7% [66.7, 74.8], tier 171.2±3.4 · T171.2% [67.8, 74.5], tier 177.3±3.9 · T277.3% [73.3, 81.2], tier 2
Gemini Decision 159.4±4.1 · T359.4% [55.3, 63.4], tier 357.8±6.9 · T257.8% [51.3, 65.1], tier 253.5±5.9 · T353.5% [47.5, 59.3], tier 380.6±6.9 · T180.6% [73.2, 87.1], tier 176.0±4.5 · T176.0% [71.5, 80.6], tier 175.0±7.8 · T175.0% [67.3, 83.0], tier 181.3±6.6 · T181.3% [74.5, 87.8], tier 171.9±8.8 · T171.9% [63.3, 80.8], tier 177.7±3.4 · T277.7% [74.2, 81.0], tier 281.5±5.2 · T181.5% [76.3, 86.6], tier 171.5±6.5 · T271.5% [64.8, 77.8], tier 280.0±5.3 · T280.0% [74.5, 85.1], tier 271.0±2.3 · T271.0% [68.6, 73.3], tier 271.4±3.9 · T171.4% [67.6, 75.4], tier 168.7±3.7 · T268.7% [65.0, 72.4], tier 277.5±4.0 · T177.5% [73.4, 81.4], tier 1
Microsoft Decision 158.5±4.0 · T358.5% [54.6, 62.6], tier 365.6±6.8 · T165.6% [59.0, 72.5], tier 150.6±5.8 · T350.6% [44.9, 56.6], tier 372.2±9.0 · T272.2% [63.0, 81.0], tier 278.8±4.4 · T178.8% [74.5, 83.3], tier 178.1±8.9 · T178.1% [68.5, 86.3], tier 182.3±7.3 · T182.3% [74.5, 89.1], tier 176.0±7.1 · T176.0% [69.2, 83.3], tier 170.5±4.0 · T470.5% [66.6, 74.5], tier 470.5±6.4 · T370.5% [63.8, 76.5], tier 369.5±7.3 · T269.5% [61.8, 76.3], tier 271.5±6.4 · T371.5% [65.1, 78.0], tier 369.3±2.5 · T369.3% [66.8, 71.7], tier 371.4±4.3 · T171.4% [67.1, 75.7], tier 167.5±3.9 · T267.5% [63.6, 71.4], tier 273.3±4.3 · T273.3% [69.0, 77.5], tier 2
Tier 4
Winnow58.5±3.6 · T458.5% [54.9, 62.2], tier 457.4±6.2 · T257.4% [51.5, 63.8], tier 254.4±5.1 · T354.4% [49.3, 59.5], tier 373.6±8.3 · T273.6% [64.9, 81.4], tier 269.1±4.7 · T269.1% [64.6, 74.0], tier 266.7±9.7 · T266.7% [57.0, 76.5], tier 267.7±7.7 · T267.7% [60.2, 75.5], tier 272.9±6.9 · T172.9% [66.0, 79.8], tier 173.0±3.8 · T373.0% [69.4, 76.9], tier 370.5±6.2 · T370.5% [64.0, 76.3], tier 369.5±6.2 · T269.5% [63.2, 75.6], tier 279.0±5.9 · T279.0% [73.1, 85.0], tier 266.9±2.3 · T366.9% [64.5, 69.0], tier 364.8±4.3 · T264.8% [60.5, 69.2], tier 263.8±3.7 · T263.8% [60.1, 67.5], tier 275.2±4.0 · T275.2% [71.0, 78.9], tier 2
Quyet-1.0-Large58.4±3.6 · T458.4% [54.9, 62.1], tier 461.3±5.8 · T261.3% [55.8, 67.3], tier 250.4±5.0 · T350.4% [45.3, 55.4], tier 379.2±6.9 · T179.2% [72.0, 85.9], tier 167.0±4.6 · T267.0% [62.6, 71.8], tier 266.7±8.0 · T266.7% [58.5, 74.5], tier 266.7±7.9 · T266.7% [58.7, 74.5], tier 267.7±6.9 · T167.7% [60.6, 74.5], tier 181.3±3.4 · T181.3% [78.0, 84.9], tier 181.0±5.8 · T181.0% [75.0, 86.6], tier 177.5±6.5 · T177.5% [70.6, 83.6], tier 185.5±5.6 · T185.5% [79.7, 91.0], tier 168.9±2.3 · T368.9% [66.7, 71.2], tier 369.7±4.0 · T169.7% [65.8, 73.7], tier 164.9±3.7 · T264.9% [61.1, 68.5], tier 277.5±3.9 · T277.5% [73.6, 81.4], tier 2
Rene-1 31B FP858.4±4.0 · T458.4% [54.4, 62.3], tier 455.2±6.7 · T255.2% [48.8, 62.2], tier 254.8±5.5 · T354.8% [48.9, 60.0], tier 375.0±8.0 · T275.0% [66.9, 82.9], tier 277.4±4.6 · T177.4% [72.9, 82.2], tier 177.1±8.7 · T177.1% [68.1, 85.4], tier 179.2±6.8 · T179.2% [72.2, 85.9], tier 176.0±7.7 · T176.0% [68.3, 83.7], tier 176.3±3.5 · T276.3% [72.9, 79.9], tier 279.5±6.0 · T179.5% [73.3, 85.3], tier 164.5±6.2 · T364.5% [58.2, 70.6], tier 385.0±5.0 · T185.0% [80.0, 90.0], tier 170.7±2.3 · T270.7% [68.4, 72.9], tier 270.6±4.4 · T170.6% [66.1, 74.9], tier 166.1±3.6 · T266.1% [62.6, 69.8], tier 278.7±4.1 · T178.7% [74.6, 82.7], tier 1
H2O-Lightning-4B v1.2.257.4±4.1 · T457.4% [53.3, 61.5], tier 457.4±6.5 · T257.4% [51.1, 64.1], tier 255.0±6.2 · T255.0% [48.7, 61.1], tier 265.3±8.8 · T365.3% [56.3, 73.8], tier 364.9±4.8 · T264.9% [60.3, 69.9], tier 269.8±8.5 · T169.8% [61.2, 78.1], tier 161.5±8.4 · T261.5% [53.1, 70.0], tier 263.5±7.5 · T263.5% [56.3, 71.3], tier 268.5±3.8 · T468.5% [64.8, 72.5], tier 467.5±6.8 · T367.5% [60.5, 74.0], tier 363.5±6.0 · T363.5% [57.4, 69.3], tier 374.5±6.3 · T374.5% [68.3, 80.9], tier 363.6±2.4 · T463.6% [61.1, 66.0], tier 464.9±4.3 · T264.9% [60.5, 69.2], tier 260.0±4.0 · T360.0% [56.0, 63.9], tier 367.8±4.4 · T367.8% [63.2, 72.0], tier 3
AutoJev-27B55.6±3.8 · T455.6% [51.9, 59.4], tier 452.6±6.7 · T252.6% [46.1, 59.6], tier 252.2±5.2 · T352.2% [47.0, 57.5], tier 371.5±8.2 · T271.5% [62.9, 79.4], tier 269.8±4.8 · T269.8% [65.0, 74.7], tier 266.7±8.3 · T166.7% [58.3, 75.0], tier 170.8±8.6 · T270.8% [62.2, 79.3], tier 271.9±6.9 · T171.9% [65.0, 78.7], tier 172.0±3.5 · T372.0% [68.7, 75.7], tier 371.5±6.6 · T271.5% [64.7, 78.0], tier 269.0±6.4 · T269.0% [62.2, 75.0], tier 275.5±5.6 · T275.5% [70.2, 81.3], tier 265.8±2.3 · T465.8% [63.5, 68.2], tier 463.6±4.3 · T263.6% [59.2, 67.9], tier 264.0±3.9 · T264.0% [60.2, 67.9], tier 273.0±4.0 · T273.0% [68.9, 76.9], tier 2
Tier 5
Jev-Omni53.6±3.7 · T553.6% [49.9, 57.4], tier 557.8±6.7 · T257.8% [51.3, 64.7], tier 246.1±5.0 · T346.1% [40.9, 51.0], tier 370.8±8.5 · T270.8% [62.0, 79.0], tier 268.8±4.1 · T268.8% [64.8, 73.0], tier 266.7±7.2 · T266.7% [59.6, 74.0], tier 268.8±6.6 · T268.8% [62.2, 75.5], tier 270.8±7.4 · T170.8% [63.3, 78.1], tier 173.3±3.8 · T373.3% [69.7, 77.2], tier 372.0±6.5 · T272.0% [65.5, 78.5], tier 272.5±6.5 · T272.5% [65.7, 78.7], tier 275.5±6.2 · T275.5% [69.3, 81.7], tier 265.2±2.3 · T465.2% [62.9, 67.6], tier 465.5±3.9 · T265.5% [61.7, 69.5], tier 262.5±3.4 · T362.5% [59.0, 65.9], tier 372.4±4.2 · T272.4% [68.0, 76.5], tier 2
decisio v0.8.051.9±4.0 · T551.9% [48.1, 56.0], tier 549.1±7.9 · T249.1% [41.4, 57.3], tier 253.0±5.1 · T353.0% [47.9, 58.0], tier 352.8±10.7 · T352.8% [42.1, 63.5], tier 370.5±4.7 · T270.5% [65.8, 75.2], tier 265.6±8.8 · T265.6% [56.9, 74.5], tier 270.8±7.2 · T270.8% [63.7, 78.1], tier 275.0±7.1 · T175.0% [68.0, 82.3], tier 181.2±3.3 · T181.2% [77.8, 84.5], tier 179.5±6.3 · T179.5% [73.0, 85.6], tier 178.5±6.2 · T178.5% [71.9, 84.3], tier 185.5±4.5 · T185.5% [81.1, 90.1], tier 167.9±2.3 · T367.9% [65.4, 70.1], tier 364.8±4.5 · T264.8% [60.3, 69.3], tier 267.5±3.8 · T267.5% [63.6, 71.3], tier 271.1±4.4 · T271.1% [66.8, 75.8], tier 2
Decision 2.0 Nox 4B51.2±4.1 · T551.2% [47.3, 55.5], tier 558.3±5.9 · T258.3% [52.6, 64.3], tier 245.2±5.7 · T345.2% [39.5, 50.8], tier 359.0±10.1 · T359.0% [48.7, 68.9], tier 359.7±4.2 · T359.7% [55.6, 63.9], tier 362.5±7.1 · T262.5% [55.3, 69.6], tier 258.3±8.3 · T258.3% [50.0, 66.7], tier 258.3±6.7 · T258.3% [52.0, 65.3], tier 256.3±3.8 · T656.3% [52.6, 60.3], tier 658.5±6.8 · T458.5% [51.7, 65.4], tier 454.5±6.3 · T454.5% [48.3, 61.0], tier 456.0±6.4 · T556.0% [49.5, 62.4], tier 555.8±2.3 · T755.8% [53.4, 58.0], tier 759.8±3.8 · T359.8% [56.0, 63.7], tier 352.7±3.8 · T452.7% [49.0, 56.7], tier 457.8±4.4 · T557.8% [53.1, 62.0], tier 5
Cygnet51.1±4.0 · T551.1% [47.2, 55.2], tier 552.2±6.4 · T252.2% [45.9, 58.7], tier 243.5±5.4 · T443.5% [38.1, 48.9], tier 473.6±8.2 · T273.6% [64.9, 81.4], tier 267.7±4.3 · T267.7% [63.4, 72.0], tier 266.7±7.6 · T166.7% [58.7, 74.0], tier 164.6±7.2 · T264.6% [57.5, 71.9], tier 271.9±6.7 · T171.9% [65.3, 78.7], tier 165.8±3.9 · T465.8% [62.0, 69.8], tier 460.0±6.9 · T460.0% [52.8, 66.7], tier 462.5±6.0 · T362.5% [56.5, 68.6], tier 375.0±6.5 · T275.0% [68.5, 81.3], tier 261.5±2.3 · T561.5% [59.2, 63.8], tier 559.6±4.3 · T359.6% [55.4, 64.0], tier 356.9±3.6 · T456.9% [53.2, 60.4], tier 473.5±4.1 · T273.5% [69.3, 77.4], tier 2
decider-4b v250.0±4.3 · T550.0% [45.8, 54.3], tier 553.0±6.9 · T253.0% [46.2, 60.1], tier 245.0±5.9 · T345.0% [39.2, 51.0], tier 361.1±9.7 · T361.1% [51.3, 70.6], tier 358.7±4.5 · T358.7% [54.2, 63.3], tier 359.4±9.0 · T259.4% [50.0, 68.1], tier 254.2±5.5 · T354.2% [48.9, 60.0], tier 362.5±8.3 · T262.5% [54.2, 70.8], tier 258.3±3.8 · T558.3% [54.5, 62.0], tier 559.5±6.9 · T459.5% [52.7, 66.5], tier 454.5±6.5 · T454.5% [48.0, 60.9], tier 461.0±6.1 · T461.0% [54.7, 67.0], tier 455.7±2.5 · T755.7% [53.2, 58.2], tier 757.3±4.4 · T357.3% [53.0, 61.9], tier 351.2±3.5 · T551.2% [47.9, 54.8], tier 561.5±4.7 · T461.5% [56.8, 66.1], tier 4
JevK549.8±3.8 · T549.8% [46.0, 53.7], tier 557.4±5.5 · T257.4% [52.0, 63.0], tier 243.7±5.5 · T443.7% [38.2, 49.2], tier 456.9±9.0 · T356.9% [47.7, 65.7], tier 355.2±3.6 · T355.2% [51.7, 58.9], tier 358.3±6.4 · T258.3% [52.0, 64.9], tier 250.0±6.6 · T350.0% [43.3, 56.5], tier 357.3±4.9 · T257.3% [53.0, 62.8], tier 257.7±3.6 · T657.7% [54.1, 61.4], tier 655.0±7.0 · T455.0% [47.7, 61.8], tier 454.5±5.9 · T454.5% [48.5, 60.2], tier 463.5±6.0 · T463.5% [57.6, 69.7], tier 454.2±2.2 · T754.2% [52.0, 56.4], tier 756.9±3.8 · T356.9% [53.4, 60.9], tier 349.4±3.4 · T549.4% [46.0, 52.8], tier 559.3±3.8 · T459.3% [55.4, 63.0], tier 4
DiffusionGemma-Jev48.7±4.3 · T548.7% [44.5, 53.0], tier 547.4±6.1 · T347.4% [41.4, 53.6], tier 343.0±6.3 · T443.0% [36.6, 49.2], tier 468.8±9.0 · T268.8% [59.1, 77.1], tier 264.2±4.6 · T264.2% [59.8, 69.0], tier 266.7±7.2 · T166.7% [59.6, 74.0], tier 163.5±7.8 · T263.5% [56.1, 71.7], tier 262.5±7.4 · T262.5% [55.3, 70.2], tier 269.5±3.8 · T469.5% [65.7, 73.3], tier 470.5±6.8 · T270.5% [63.8, 77.4], tier 263.0±6.6 · T363.0% [56.3, 69.5], tier 375.0±5.6 · T275.0% [69.5, 80.6], tier 260.8±2.5 · T560.8% [58.3, 63.2], tier 561.5±4.0 · T261.5% [57.5, 65.6], tier 256.5±3.9 · T456.5% [52.6, 60.4], tier 468.8±4.4 · T368.8% [64.3, 73.1], tier 3
Tier 6
Clef Omni48.4±4.0 · T648.4% [44.6, 52.5], tier 657.0±6.1 · T257.0% [51.1, 63.2], tier 239.4±5.6 · T439.4% [33.7, 45.0], tier 463.9±8.5 · T363.9% [55.2, 72.2], tier 355.6±4.3 · T355.6% [51.4, 60.0], tier 359.4±7.9 · T259.4% [52.0, 67.7], tier 246.9±7.5 · T346.9% [39.1, 54.1], tier 360.4±5.6 · T260.4% [55.1, 66.3], tier 262.8±3.7 · T562.8% [59.1, 66.5], tier 568.5±6.8 · T368.5% [61.3, 74.9], tier 353.5±5.9 · T453.5% [47.4, 59.3], tier 466.5±6.5 · T366.5% [60.0, 73.1], tier 355.6±2.3 · T755.6% [53.3, 57.9], tier 761.6±4.0 · T261.6% [57.6, 65.7], tier 246.6±3.7 · T646.6% [42.9, 50.2], tier 663.6±4.0 · T363.6% [59.5, 67.5], tier 3
Hopper 12B48.3±4.0 · T648.3% [44.5, 52.5], tier 657.0±6.5 · T257.0% [50.8, 63.8], tier 238.0±5.1 · T538.0% [32.9, 43.2], tier 567.4±8.4 · T267.4% [58.7, 75.5], tier 262.5±4.6 · T262.5% [57.9, 67.1], tier 267.7±8.8 · T167.7% [58.5, 76.1], tier 157.3±7.7 · T257.3% [50.0, 65.3], tier 262.5±7.5 · T262.5% [55.2, 70.2], tier 265.8±3.9 · T465.8% [62.0, 69.8], tier 465.0±6.9 · T365.0% [57.8, 71.6], tier 357.0±6.3 · T357.0% [50.5, 63.2], tier 375.5±6.0 · T275.5% [69.5, 81.5], tier 258.9±2.4 · T658.9% [56.4, 61.2], tier 663.2±4.1 · T263.2% [59.1, 67.4], tier 250.8±3.6 · T550.8% [47.1, 54.3], tier 568.5±4.2 · T368.5% [64.2, 72.6], tier 3
Julia-147.4±3.6 · T647.4% [43.7, 50.9], tier 647.8±6.5 · T247.8% [41.3, 54.3], tier 248.3±4.5 · T348.3% [43.6, 52.6], tier 343.8±8.8 · T343.8% [34.8, 52.3], tier 350.3±3.3 · T450.3% [46.9, 53.5], tier 451.0±6.3 · T351.0% [44.8, 57.5], tier 353.1±7.0 · T353.1% [46.1, 60.0], tier 346.9±4.2 · T346.9% [42.5, 51.0], tier 350.0±3.6 · T750.0% [46.4, 53.5], tier 753.0±5.6 · T453.0% [47.7, 58.8], tier 449.5±6.3 · T449.5% [43.4, 56.1], tier 447.5±6.9 · T647.5% [40.5, 54.3], tier 649.2±2.0 · T849.2% [47.2, 51.3], tier 850.6±3.5 · T450.6% [47.1, 54.1], tier 450.3±3.4 · T550.3% [46.8, 53.6], tier 546.0±4.1 · T646.0% [42.0, 50.1], tier 6
Clef Flash47.1±4.4 · T647.1% [42.8, 51.6], tier 660.9±6.9 · T260.9% [54.0, 67.8], tier 235.9±6.0 · T535.9% [29.8, 41.8], tier 561.1±9.6 · T361.1% [51.1, 70.3], tier 363.2±4.1 · T263.2% [59.3, 67.4], tier 268.8±7.9 · T168.8% [60.6, 76.5], tier 159.4±7.1 · T259.4% [52.2, 66.3], tier 261.5±6.4 · T261.5% [55.3, 68.1], tier 257.7±4.0 · T657.7% [53.8, 61.8], tier 658.0±7.0 · T458.0% [51.0, 65.0], tier 455.0±6.0 · T455.0% [49.0, 61.1], tier 460.0±7.1 · T460.0% [52.9, 67.0], tier 456.0±2.3 · T656.0% [53.7, 58.3], tier 662.5±4.1 · T262.5% [58.5, 66.7], tier 250.1±3.8 · T550.1% [46.4, 54.0], tier 560.9±4.4 · T460.9% [56.2, 65.0], tier 4
Decision 2.0 Lux 9B45.4±4.1 · T645.4% [41.6, 49.7], tier 652.6±6.6 · T252.6% [46.2, 59.4], tier 237.0±5.6 · T537.0% [31.1, 42.4], tier 561.1±9.3 · T361.1% [51.4, 70.1], tier 362.2±4.0 · T262.2% [58.2, 66.3], tier 264.6±6.8 · T264.6% [57.6, 71.3], tier 259.4±6.7 · T259.4% [52.9, 66.3], tier 262.5±6.9 · T262.5% [56.0, 69.8], tier 261.0±3.9 · T561.0% [57.2, 65.0], tier 559.0±6.9 · T459.0% [52.0, 65.8], tier 459.5±6.6 · T359.5% [53.1, 66.2], tier 364.5±6.8 · T464.5% [57.4, 71.0], tier 456.2±2.3 · T656.2% [53.9, 58.5], tier 658.7±4.0 · T358.7% [54.8, 62.8], tier 351.9±3.6 · T551.9% [48.5, 55.7], tier 562.7±4.6 · T462.7% [58.1, 67.2], tier 4
Decision 2.0 Sol 2B45.1±4.4 · T645.1% [40.8, 49.5], tier 661.3±6.8 · T261.3% [54.4, 67.9], tier 234.6±6.0 · T534.6% [28.6, 40.6], tier 552.8±9.7 · T352.8% [42.6, 61.9], tier 350.3±2.9 · T450.3% [47.6, 53.4], tier 453.1±5.9 · T253.1% [46.9, 58.7], tier 246.9±4.4 · T346.9% [42.4, 51.1], tier 351.0±3.6 · T351.0% [47.9, 55.1], tier 348.5±3.8 · T848.5% [44.7, 52.3], tier 851.0±6.5 · T551.0% [44.5, 57.5], tier 546.5±6.9 · T446.5% [39.8, 53.6], tier 448.0±6.4 · T648.0% [41.7, 54.5], tier 648.0±2.3 · T848.0% [45.7, 50.2], tier 855.1±3.6 · T355.1% [51.5, 58.8], tier 342.6±3.4 · T642.6% [39.3, 46.1], tier 650.6±4.0 · T650.6% [46.5, 54.6], tier 6
Decision 2.0 Eos 0.8B44.8±4.0 · T644.8% [41.0, 49.0], tier 656.5±5.9 · T256.5% [50.6, 62.4], tier 236.1±5.9 · T536.1% [30.2, 41.9], tier 554.2±8.5 · T354.2% [45.3, 62.3], tier 352.4±2.8 · T452.4% [49.6, 55.2], tier 454.2±4.5 · T254.2% [50.0, 59.0], tier 250.0±6.2 · T350.0% [43.8, 56.1], tier 353.1±3.6 · T253.1% [50.0, 57.1], tier 249.7±3.7 · T749.7% [45.9, 53.4], tier 751.5±6.4 · T551.5% [45.1, 57.8], tier 548.0±6.5 · T448.0% [41.5, 54.5], tier 449.5±6.1 · T649.5% [43.4, 55.6], tier 649.0±2.1 · T849.0% [46.9, 51.1], tier 854.1±3.4 · T354.1% [50.7, 57.5], tier 344.7±3.5 · T644.7% [41.3, 48.2], tier 652.3±3.8 · T652.3% [48.5, 56.0], tier 6
Decision 2.0 Kai 0.6B43.5±4.3 · T643.5% [39.3, 48.0], tier 650.9±6.9 · T250.9% [43.6, 57.5], tier 235.0±5.9 · T535.0% [29.1, 40.9], tier 559.0±8.8 · T359.0% [50.0, 67.7], tier 350.3±2.5 · T450.3% [47.9, 52.8], tier 452.1±5.1 · T352.1% [46.9, 57.0], tier 346.9±4.3 · T346.9% [42.5, 51.1], tier 352.1±2.7 · T252.1% [50.0, 55.3], tier 248.3±4.0 · T848.3% [44.3, 52.3], tier 850.0±7.0 · T550.0% [43.1, 57.1], tier 545.0±6.6 · T545.0% [38.5, 51.6], tier 550.0±7.1 · T550.0% [42.9, 57.2], tier 547.4±2.1 · T847.4% [45.3, 49.5], tier 851.0±3.7 · T451.0% [47.3, 54.6], tier 442.3±3.1 · T642.3% [39.2, 45.4], tier 653.7±3.9 · T553.7% [49.6, 57.5], tier 5
Tier 7
Lev 4B42.1±4.0 · T742.1% [38.1, 46.2], tier 755.6±6.2 · T255.6% [49.6, 61.9], tier 230.2±5.6 · T630.2% [24.5, 35.6], tier 658.3±10.0 · T358.3% [48.0, 68.0], tier 353.5±2.8 · T353.5% [50.7, 56.3], tier 353.1±4.5 · T253.1% [49.0, 58.0], tier 253.1±5.3 · T353.1% [47.9, 58.5], tier 354.2±3.7 · T254.2% [51.0, 58.5], tier 252.7±3.6 · T752.7% [49.0, 56.3], tier 750.5±6.3 · T550.5% [44.3, 56.9], tier 553.0±6.2 · T453.0% [46.9, 59.3], tier 454.5±6.4 · T554.5% [48.0, 60.9], tier 549.4±2.1 · T849.4% [47.4, 51.5], tier 853.1±3.3 · T453.1% [50.0, 56.5], tier 445.5±3.3 · T645.5% [42.3, 48.8], tier 655.7±4.1 · T555.7% [51.5, 59.8], tier 5

Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).

Jev

Provider
TypeSafe
Weights
Closed (API)
Size
Not disclosed
Licence
Proprietary
TaskScore [95% interval]TierLatency p50Cost / 1k
All three jobs, core66.8% [63.2, 70.2]2321 ms (Hosted API)$0.39
LLM-as-a-judge, core62.0% [56.1, 68.3]2411 ms (Hosted API)$0.49
Search, core68.2% [63.0, 73.1]1, tied319 ms (Hosted API)$0.32
Scenario judge, core69.9% [61.0, 78.0]2273 ms (Hosted API)$0.43
All three jobs, long conversations73.6% [68.8, 78.3]1, tied453 ms (Hosted API)$0.82
LLM-as-a-judge, long conversations72.9% [64.5, 80.8]1, tied447 ms (Hosted API)$0.85
Search, long conversations73.6% [65.0, 82.0]2463 ms (Hosted API)$0.84
Scenario judge, long conversations74.3% [66.7, 81.7]1, tied454 ms (Hosted API)$0.77
All three jobs, hard cases74.8% [71.3, 78.2]3270 ms (Hosted API)$0.086
LLM-as-a-judge, hard cases76.7% [70.6, 82.5]2293 ms (Hosted API)$0.11
Search, hard cases73.0% [66.7, 79.0]2262 ms (Hosted API)$0.11
Scenario judge, hard cases74.7% [68.7, 80.3]3261 ms (Hosted API)$0.040
All three jobs, all three sets71.7% [69.3, 74.0]2348 ms (Hosted API)$0.43
LLM-as-a-judge, all three sets70.5% [66.5, 74.4]1, tied384 ms (Hosted API)$0.48
Search, all three sets71.6% [67.6, 75.5]1, tied348 ms (Hosted API)$0.42
Scenario judge, all three sets73.0% [68.5, 77.3]2329 ms (Hosted API)$0.42
  • Results are for Jev 1.14.0, called on 15 October 2026, three passes over every check; its score is the mean of the passes. The passes sent more calls at once than the earlier runs, so its times are not from the same setup as the other models’. Every other model’s answers are as before.
How we measured

Scores carry a 95% bootstrap interval (2,000 resamples, the checks built from one scenario resampled together). Two models are tied when the 95% interval of their paired difference on the same checks holds zero.

Tie tiers

Models in one tier can't be separated by the tie test (the 95% interval of the paired difference on the same cases holds zero). Tiers are computed in the export over every model; filters on this page never change them.

Contamination

No check was ever published, so no model can have trained on them. Core and long conversations are real-world based; the hard cases are synthetic, generated for this benchmark from made-up data, with every label computed from that data. The page shows aggregates only: no check and no check text.

Latency and hardware

  • Hosted API: Jev, OpenAI Decisions API, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1 and Gemini Decision 1, called as hosted APIs from a laptop, one call per check; the time includes the network
  • H100: Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8 and H2O-Lightning-4B v1.2.2, open weights on one rented H100; the server’s own time per check, 30 checks one at a time
  • L4: Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B and Decision 2.0 Kai 0.6B, open weights on one rented L4; the server’s own time per check, 30 checks one at a time
  • Server CPU: Julia-1, served on the GPU box’s CPU; the server’s own time per check, 30 checks one at a time

Latencies are only compared within one hardware tier.

Datasets

  • All three jobs, core (834 test items): real-world based labelled checks, never published (private, aggregates only)
  • LLM-as-a-judge, core (230 test items): real-world based labelled checks, never published (private, aggregates only)
  • Search, core (460 test items): real-world based labelled checks, never published (private, aggregates only)
  • Scenario judge, core (144 test items): real-world based labelled checks, never published (private, aggregates only)
  • All three jobs, long conversations (288 test items): real-world based long sessions, never published (private, aggregates only)
  • LLM-as-a-judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
  • Search, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
  • Scenario judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
  • All three jobs, hard cases (600 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
  • LLM-as-a-judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
  • Search, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
  • Scenario judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
  • All three jobs, all three sets (1,722 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
  • LLM-as-a-judge, all three sets (526 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
  • Search, all three sets (756 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
  • Scenario judge, all three sets (440 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
  • An error, a reply with no verdict, or an inconclusive Scenario verdict counts as wrong.
  • All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
  • Scores count answers on texts cut to fit the model’s window. For 3 models some texts were still too long after the cut LangWatch uses in production, so they were cut further until they fit. Counted as errors instead, which is what production would return, their scores are lower: AutoJev-27B, 161 core and 22 long conversation checks (55.6% becomes 45.9% and 69.8% becomes 64.6%); Lev 4B, 155 core and 16 long conversation checks (42.1% becomes 33.8% and 53.5% becomes 51.0%); Quyet-1.0-Large, 260 core and 186 long conversation checks (58.4% becomes 40.9% and 67.0% becomes 23.6%).
  • Latency is the median time per check, compared only within one hardware tier. Hosted APIs were timed from a laptop on every check, so their time includes the network. Open-weights models were timed on the GPU box by the server itself, 30 checks one at a time, one number for every job and set.
  • The 23 open-weights models ran in bf16 on rented L4 and H100 GPUs (Winnow in Q8_0 GGUF, decisio v0.8.0 in FP8, deck-31B in FP8 and Rene-1 31B FP8 in FP8). In the timing run, Julia-1 was served on the box’s CPU.
  • Jev: Results are for Jev 1.14.0, called on 15 October 2026, three passes over every check; its score is the mean of the passes. The passes sent more calls at once than the earlier runs, so its times are not from the same setup as the other models’. Every other model’s answers are as before.
  • AutoJev-27B: The same weights as Perplexity’s pplx-decider v1 27B, file for file, so the two are one model here.
  • Winnow: Run from its Q8_0 GGUF weights, the precision its authors recommend.
  • Cygnet: A recipe, not a trained model: a letter readout on stock Gemma 4 12B IT. Run without its calibration temperature, which rescales probabilities but cannot flip a yes or no at the 0.5 threshold.
  • decider-4b v2: Served with a smaller CUDA graph token budget than its default, a capacity setting that does not change answers.
  • JevK5: Takes an 8k token state.
  • Lev 4B: Takes an 8k token state.
  • Julia-1: Served on the box’s CPU, so its latency is CPU time.
  • Quyet-1.0-Large: Reads at most 6,000 tokens of a conversation, so most core and long conversation texts were cut to fit.
  • decisio v0.8.0: A serving recipe on stock Gemma 4 31B. Refused 146 core checks whose text carries inline image data; each counts as wrong.
  • deck-31B: A recipe, not a trained model: a letter readout on stock Gemma 4 31B, run in FP8.
  • Rene-1 31B FP8: Ran on one to three GPUs at once, so its cost per 1,000 checks is approximate.
  • H2O-Lightning-4B v1.2.2: Version 1.2.2, the release its card tells users to download; the public JevBench board ranks version 1.1, which has different weights.
  • Mercury Decide: Hard cases through Inception’s own API; core and long conversations through OpenRouter, with zero data retention.
  • Clef: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, after Cloudflare’s faster serving.
  • Clef Flash: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, at its new price. The hosted service reads at most 24,000 tokens of a check and cuts longer ones without an error.
  • GLiDE: A hosted “thinking” decision model: an uncertain check can take extra time. Refused 0 core checks that are over its context window; each counts as wrong.
  • Microsoft Decision 1: Called through OpenRouter, with zero data retention. Reads at most 32,768 tokens: refused 0 core checks over that limit, each counts as wrong.
  • Gemini Decision 1: Google’s typed decision API: it answers a yes or no question with a probability and no text. Called on Google’s own API, paid tier, at $0.10 per 1M input tokens and no output charge.
  • pplx-decider v1.1 27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.02 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.51 on core, $0.51 on long conversations and $0.27 on hard cases per 1,000 checks.
  • AutoJev-27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.04 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.45 on core, $0.51 on long conversations and $0.26 on hard cases per 1,000 checks.
  • Cost per 1,000 checks: hosted APIs at the price recorded on the run, pplx-decider v1.1 27B and AutoJev-27B at Perplexity’s hosted API price. Other open-weights models at GPU rent (€2.87 and €3.31 per GPU hour, in dollars at 1.149 per euro) times the GPU time each check took on that set, shared with the other models on the same card; on hard cases, the box’s median time per check, one check at a time.
  • Long-context card: Jev 1.14 with the TypeSafe-Danger-Zone-Long-Context request header against Jev 1.14 without it, 3 passes each, on the same checks. A gain is claimed only when the 95% interval of the paired difference lies above zero. Long LLM-as-a-judge checks: 96 from the long-conversation set. Invented conversations: 200 checks LangWatch generated (not published, no customer text), 100% of them cut to fit without the header and 0% with it. Cost and the slowest times (95th percentile) are from the same calls on the long-conversation set.

Release jev-class-2026-11-14, integrity sha256-leaves-v1.