OpenAI Decisions API beats Jev on real-world checks
LangWatch LLM-as-a-Judge Bench evaluates model capabilities in doing evals, agent simulations and trace search, our specialty. Core and long conversations are real-world based, hard cases synthetic; none is public, to avoid benchmaxing and measure true capability for our customers.
On real conversations OpenAI Decisions API and Mercury Decide score highest, for different reasons. The OpenAI Decisions API misses fewer real problems, above all facts the agent made up and mistakes made earlier in the conversation; Mercury Decide less often flags a response that is fine.
When an agent says it did something it never did, such as opening a ticket, no decision model reliably notices: the best catches only about two in five of these, and an LLM does little better.
What each model catches
Model
Made-up facts: The reply states a price, a setting or a step that no tool or document returned.
A claimed action that never happened: The agent says it opened a ticket or sent a message, but no tool call did.
A requirement only half met: The agent declines an off-topic question but never steers back to the task.
Leaves a good reply alone: Nothing is wrong and the model says so. A low share means false alarms.
The reply states a price, a setting or a step that no tool or document returned.
A claimed action that never happened
The agent says it opened a ticket or sent a message, but no tool call did.
A requirement only half met
The agent declines an off-topic question but never steers back to the task.
Leaves a good reply alone
Nothing is wrong and the model says so. A low share means false alarms.
The share of checks each model gets right; deeper orange is a higher share. Models are in the order of the chart above. A long input is compacted to fit a model’s window, the same way for every model; a model not yet rerun that way counts only the checks it read in full, and is not listed without enough of them for every column.
One job at a time
All three jobs, core: Accuracy, higher is better
Vega 27B Fine-Tuned (LangWatch)Vega 27B Fine-Tuned (LangWatch): 73.0% [69.3, 76.6]. Not ranked.Vega 27B Fine-Tuned (LangWatch): 73.0% [69.3, 76.6]. Not ranked.73.0%, interval 73.0% [69.3, 76.6], not ranked
OpusClaude Opus 5.5: 90.0%, ~$75, for reference, not ranked.
SolGPT-6.1 Sol: 89.0%, ~$36, for reference, not ranked.
LunaGPT-6 Luna: 78.1%, $1.22, for reference, not ranked.
FlashGemini 3.8 Flash: 83.6%, $16, for reference, not ranked.
ArgonGemini 4 Argon: 82.9%, $36, for reference, not ranked.
HydrogenGemini 4 Hydrogen: 88.5%, $34, for reference, not ranked.
$0.01$0.1$1$10$100← Cost per 1,000 decisions, USD (log scale). Lower is betterHigher is better ↑
Blue LLMs are not ranked. Opus and Sol (hollow): an estimate at API list price. LLM cost is one figure over all three sets.
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~). Not plotted (no cost measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Every hardware tier on one plot, with each point’s tier beside its name and in its tooltip. The hosted APIs were timed from a laptop over the internet, so their time includes the network; the rest were timed on the GPU box itself. Not plotted (no latency measured): Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Hosted API. Not plotted (no latency on Hosted API measured): Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: H100. Not plotted (no latency on H100 measured): Jev, OpenAI Decisions API, Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Julia-1, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: L4. Not plotted (no latency on L4 measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Julia-1, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Dashed line: the Pareto frontier (no other model is both better and faster). Hollow mark: an estimate (~). Latency is only compared within one hardware tier: Server CPU. Not plotted (no latency on Server CPU measured): Jev, OpenAI Decisions API, Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Winnow, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B, Decision 2.0 Kai 0.6B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8, H2O-Lightning-4B v1.2.2, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1, Gemini Decision 1, Vega 27B Fine-Tuned (LangWatch).
Jev alternatives, every job
Every visible model on every task, sorted by All three jobs, core. Value with half its interval; bands are tie tiers.
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Results are for Jev 1.14.0, called on 15 October 2026, three passes over every check; its score is the mean of the passes. The passes sent more calls at once than the earlier runs, so its times are not from the same setup as the other models’. Every other model’s answers are as before.
How we measured
Scores carry a 95% bootstrap interval (2,000 resamples, the checks built from one scenario resampled together). Two models are tied when the 95% interval of their paired difference on the same checks holds zero.
Tie tiers
Models in one tier can't be separated by the tie test (the 95% interval of the paired difference on the same cases holds zero). Tiers are computed in the export over every model; filters on this page never change them.
Contamination
No check was ever published, so no model can have trained on them. Core and long conversations are real-world based; the hard cases are synthetic, generated for this benchmark from made-up data, with every label computed from that data. The page shows aggregates only: no check and no check text.
Latency and hardware
Hosted API: Jev, OpenAI Decisions API, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni, Microsoft Decision 1 and Gemini Decision 1, called as hosted APIs from a laptop, one call per check; the time includes the network
H100: Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8 and H2O-Lightning-4B v1.2.2, open weights on one rented H100; the server’s own time per check, 30 checks one at a time
L4: Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B and Decision 2.0 Kai 0.6B, open weights on one rented L4; the server’s own time per check, 30 checks one at a time
Server CPU: Julia-1, served on the GPU box’s CPU; the server’s own time per check, 30 checks one at a time
Latencies are only compared within one hardware tier.
Datasets
All three jobs, core (834 test items): real-world based labelled checks, never published (private, aggregates only)
LLM-as-a-judge, core (230 test items): real-world based labelled checks, never published (private, aggregates only)
Search, core (460 test items): real-world based labelled checks, never published (private, aggregates only)
Scenario judge, core (144 test items): real-world based labelled checks, never published (private, aggregates only)
All three jobs, long conversations (288 test items): real-world based long sessions, never published (private, aggregates only)
LLM-as-a-judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
Search, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
Scenario judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
All three jobs, hard cases (600 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
LLM-as-a-judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
Search, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
Scenario judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
All three jobs, all three sets (1,722 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
LLM-as-a-judge, all three sets (526 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
Search, all three sets (756 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
Scenario judge, all three sets (440 test items): the mean of the core, long conversation and hard case sets (private, aggregates only)
An error, a reply with no verdict, or an inconclusive Scenario verdict counts as wrong.
All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
Scores count answers on texts cut to fit the model’s window. For 3 models some texts were still too long after the cut LangWatch uses in production, so they were cut further until they fit. Counted as errors instead, which is what production would return, their scores are lower: AutoJev-27B, 161 core and 22 long conversation checks (55.6% becomes 45.9% and 69.8% becomes 64.6%); Lev 4B, 155 core and 16 long conversation checks (42.1% becomes 33.8% and 53.5% becomes 51.0%); Quyet-1.0-Large, 260 core and 186 long conversation checks (58.4% becomes 40.9% and 67.0% becomes 23.6%).
Latency is the median time per check, compared only within one hardware tier. Hosted APIs were timed from a laptop on every check, so their time includes the network. Open-weights models were timed on the GPU box by the server itself, 30 checks one at a time, one number for every job and set.
The 23 open-weights models ran in bf16 on rented L4 and H100 GPUs (Winnow in Q8_0 GGUF, decisio v0.8.0 in FP8, deck-31B in FP8 and Rene-1 31B FP8 in FP8). In the timing run, Julia-1 was served on the box’s CPU.
Jev: Results are for Jev 1.14.0, called on 15 October 2026, three passes over every check; its score is the mean of the passes. The passes sent more calls at once than the earlier runs, so its times are not from the same setup as the other models’. Every other model’s answers are as before.
AutoJev-27B: The same weights as Perplexity’s pplx-decider v1 27B, file for file, so the two are one model here.
Winnow: Run from its Q8_0 GGUF weights, the precision its authors recommend.
Cygnet: A recipe, not a trained model: a letter readout on stock Gemma 4 12B IT. Run without its calibration temperature, which rescales probabilities but cannot flip a yes or no at the 0.5 threshold.
decider-4b v2: Served with a smaller CUDA graph token budget than its default, a capacity setting that does not change answers.
JevK5: Takes an 8k token state.
Lev 4B: Takes an 8k token state.
Julia-1: Served on the box’s CPU, so its latency is CPU time.
Quyet-1.0-Large: Reads at most 6,000 tokens of a conversation, so most core and long conversation texts were cut to fit.
decisio v0.8.0: A serving recipe on stock Gemma 4 31B. Refused 146 core checks whose text carries inline image data; each counts as wrong.
deck-31B: A recipe, not a trained model: a letter readout on stock Gemma 4 31B, run in FP8.
Rene-1 31B FP8: Ran on one to three GPUs at once, so its cost per 1,000 checks is approximate.
H2O-Lightning-4B v1.2.2: Version 1.2.2, the release its card tells users to download; the public JevBench board ranks version 1.1, which has different weights.
Mercury Decide: Hard cases through Inception’s own API; core and long conversations through OpenRouter, with zero data retention.
Clef: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, after Cloudflare’s faster serving.
Clef Flash: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, at its new price. The hosted service reads at most 24,000 tokens of a check and cuts longer ones without an error.
GLiDE: A hosted “thinking” decision model: an uncertain check can take extra time. Refused 0 core checks that are over its context window; each counts as wrong.
Microsoft Decision 1: Called through OpenRouter, with zero data retention. Reads at most 32,768 tokens: refused 0 core checks over that limit, each counts as wrong.
Gemini Decision 1: Google’s typed decision API: it answers a yes or no question with a probability and no text. Called on Google’s own API, paid tier, at $0.10 per 1M input tokens and no output charge.
pplx-decider v1.1 27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.02 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.51 on core, $0.51 on long conversations and $0.27 on hard cases per 1,000 checks.
AutoJev-27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.04 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.45 on core, $0.51 on long conversations and $0.26 on hard cases per 1,000 checks.
Cost per 1,000 checks: hosted APIs at the price recorded on the run, pplx-decider v1.1 27B and AutoJev-27B at Perplexity’s hosted API price. Other open-weights models at GPU rent (€2.87 and €3.31 per GPU hour, in dollars at 1.149 per euro) times the GPU time each check took on that set, shared with the other models on the same card; on hard cases, the box’s median time per check, one check at a time.
Long-context card: Jev 1.14 with the TypeSafe-Danger-Zone-Long-Context request header against Jev 1.14 without it, 3 passes each, on the same checks. A gain is claimed only when the 95% interval of the paired difference lies above zero. Long LLM-as-a-judge checks: 96 from the long-conversation set. Invented conversations: 200 checks LangWatch generated (not published, no customer text), 100% of them cut to fit without the header and 0% with it. Cost and the slowest times (95th percentile) are from the same calls on the long-conversation set.