Frontier LLMs beat Jev on 9 of 11 tasks

1. 1

   Same tasks, same sets

   The 11 use cases of the Jev benchmark, with the same frozen dev and test splits (up to 1,000 test items each). The open models on the same sets.

2. 2

   One prompt per decision

   Each LLM got the task and its options in the system message and the item last, and answered with one label through a JSON schema. The wording was frozen on the 60‑item dev split before the test run. Reasoning was off, or at the lowest setting each model accepts; some models still reason briefly at their lowest setting.

3. 3

   Label‑only scores

   The frontier LLMs answer with one label, not a probability, so threshold-free metrics (AUROC, catch rate at a fixed false-alarm rate) cannot be computed for them. On moderation, PII and prompt injection every model is therefore compared on balanced accuracy, computed the same way for all: Jev at the decision threshold frozen on the dev split, each LLM at its label.

4. 4

   Intervals and the tie test

   Scores carry a 95% percentile bootstrap interval over items (2,000 resamples on the use cases). The tie test is a paired McNemar test on accuracy, or a paired bootstrap of the difference in balanced accuracy, on the same items for every model; an empty answer counts as wrong.

5. 5

   Latency

   Frontier LLMs: each call timed from Denmark through the LangWatch AI Gateway, 6 at a time, network and gateway included (p50 1,030 ms to 2,539 ms by task and model). Jev: approximate server‑side time (~, p50 ~59 ms to ~80 ms). The two bases differ, so the page names no fastest model across them.

6. 6

   Cost

   Recorded token usage of the test run at each vendor’s public list price, cached input included where the provider applied it; Jev at its list price (input tokens only, output free). Shown per prompt; the export also carries it per 1,000 decisions. Jev is the cheapest on nine of 11 tasks; GPT-6 Luna costs less on off-topic and routing, 77 intents.

7. 7

   Through the gateway

   Every frontier LLM call went through the LangWatch AI Gateway, which logged the cost and latency of each call, with prompt caching on the shared part of long prompts.

8. 8

   Coverage

   A model is ranked when it answered at least 95% of a task’s items; an empty answer counts as wrong.

What is left out

• No flip rate for the frontier LLMs: their test runs were not repeated, apart from GPT-6 Luna on two tasks (in the caveats).

Datasets and provenance

Tasks from public datasets, each with its licence. Two tasks are scored without a source licensed for non‑commercial use.

• JailbreaksOverTime: Piet et al. 2025, CC-BY-4.0.
• Nemotron-PII: NVIDIA, CC-BY-4.0 (attribution requested when results are published).
• Aegis AI Content Safety 2.0: NVIDIA, CC-BY-4.0.
• Banking77: Casanueva et al. 2020, CC-BY-4.0.
• CLINC150: Larson et al. 2019, CC-BY-3.0. MASSIVE: Amazon, CC-BY-4.0. SNIPS: Sonos, CC0-1.0.
• Shopping Queries Dataset (ESCI): Reddy et al. 2022, Apache-2.0.
• Berkeley Function Calling Leaderboard v3 live: Apache-2.0.
• CFPB consumer complaint narratives: US government public domain (BEE-spoke copy, CC0-1.0).
• HaluEval qa and summarization: Li et al. 2023 (HotpotQA CC-BY-SA-4.0, CNN/DailyMail).
• Banking77: Casanueva et al. 2020, CC-BY-4.0.

Footnotes

1. §Jev latency: Jev is a hosted API measured from Denmark. For p50 we measured the network round trip to its servers in the same window (71 requests, p50 208 ms) and subtracted it from our measured p50. The network tail could not be subtracted reliably, so p95 is Jev’s own server‑reported p95 (95 to 114 ms by task). Both are approximate, and neither is on the frontier LLMs’ basis, which includes the network and the gateway.

Provenance

Release 2026-09-26.2, benchmark commit 2386153. All 66 runs behind these numbers were made from a working tree with uncommitted harness changes, so the exact code of those runs is not reproducible from a commit alone.

Spot something outdated? Tell us.