Open models caught up with Jev

1. 1

   Frozen splits

   Each use case has a 60‑item dev split and a test split of up to 1,000 items, frozen with checksums before any model ran.

2. 2

   Wording chosen on dev

   Every model got each task in two or three wordings; the wording and the decision threshold were picked on the dev split, once, then frozen.

3. 3

   Intervals and the tie test

   Scores carry a 95% percentile bootstrap interval over items (2,000 resamples on the use cases). The general suites use 10,000 resamples, clustered so the variants of one item move together. The tie test is a paired McNemar test on accuracy, or a paired bootstrap of the metric difference, on the items every model answered.

4. 4

   Latency

   Open models ran one at a time on a laptop GPU, an L4 or an H100, so latency compares only on the same GPU; every p95 names its GPU. Jev’s latency is measured from a client and includes the network. Cost is not shown on this release.

5. 5

   Long inputs

   Inputs over each set’s length limit (2,500 to 12,000 characters) were removed for every model alike before sampling. Inputs a model cuts at its own token window stay in the score.

6. 6

   Contamination

   Each model and task pair was audited against the model’s published training, selection and calibration data. Grayed rows trained on the test items or on the same dataset; footnoted rows trained on a related task. Models are marked on what their authors disclosed; no declaration is not the same as not trained on it.

The contamination rule

Grayed: trained on the test items or on the same dataset for at least a third of the category. Footnoted: trained on a related task. Unknown: training data not disclosed. Grayed entries are shown for reference and never ranked as the winner. Grayed rows are shown for reference, never ranked.

Jev’s training data is not disclosed, and every open model starts from a pretrained base whose corpus is not itemised. An unmarked row means no known contamination, not proof that there is none.

Shisa DE-1 declares no training data and is grayed on no task; openJev Verdict 1.4 disclosed its data and is grayed on 4 of 14.

Where a grayed model has the best open score, counted:

• Moderation: AutoJev-27B* (trained on the dataset behind 33% of the test items) scores 90.6%, 0.4 points ahead of Jev.
• Off-topic: AutoJev-27B* (trained on the dataset behind 33% of the test items) scores 92.7%, 0.7 points behind Jev.
• Routing, 20 intents: Kev-4B* (trained on the dataset behind 100% of the test items) scores 93.0%, 3.9 points ahead of Jev.
• Routing, 77 intents: Kev-4B* (trained on the dataset behind 100% of the test items) scores 85.0%, 5.4 points ahead of Jev.

What is left out

• JevBench public is left out: it is the set open models are tuned against.

What the data‑quality audit found

No trivial baseline beats the best open model on any of the 11 tasks that have one. The notes below say what a better set would change.

• Prompt injection. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 31.0%; Bag of words (TF-IDF) 72.6%. SPML is separable by length alone (length-only AUROC 0.998): its benign items are short questions and its injections long persona prompts. Without it, the best open model changes. Without spml (750 items): Jev 92.5%, Eikos-27B 92.3%, Shisa DE-1 92.3%.
• Moderation. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 57.6%; Bag of words (TF-IDF) 76.3%.
• PII. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 3.2%; Bag of words (TF-IDF) 77.2%; Six regexes 68.7%. The gretel and nemotron negatives carry a redaction stand-in ('on file', 'the customer') that no positive has, so a bag-of-words classifier separates them perfectly (AUROC 1.000). The number without them is on privy alone, the one source without that artefact, where the gap between Jev and the best open model is a fraction of the headline gap. Without gretel and nemotron (333 items): Kev-9B 91.6%, AutoJev-27B 90.4%, Kev-4B 88.6%.
• RAG faithfulness. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 67.8%; Bag of words (TF-IDF) 54.6%; String-overlap heuristic 68.0%. HaluEval-qa is a length and style detector: 92.8% of its hallucinated answers end in a full stop against 1.2% of the correct ones, and a five-feature string-overlap heuristic (AUROC 0.977) beats Jev (0.934) on it. The number without it is on HaluEval summarization alone. Without halueval-dialogue and halueval-qa (333 items): Eikos-27B 79.0%, Shisa DE-1 78.6%, Kev-9B 76.6%.
• Off-topic. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 50.8%; Bag of words (TF-IDF) 83.5%. SNIPS is separable by keywords (TF-IDF AUROC 0.999): it holds only two out-of-scope intent types. Without snips (667 items): Jev 91.9%, Kev-9B 87.3%, Shisa DE-1 84.9%.
• Routing, 20 intents. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 3.1%; Bag of words (TF-IDF) 66.7%.
• Routing, 77 intents. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 4.6%; Bag of words (TF-IDF) 49.2%.
• Tool routing. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 48.3%; Bag of words (TF-IDF) 49.5%; Option count only 49.2%; Lexical matcher (unsupervised) 63.2%; Option count only (none-of-these detection, AUROC) 0.793. BFCL's irrelevance half marks a request irrelevant when a call cannot be made (a required argument is missing), while this task asks which tool should handle it: 20.7% of that source is mislabelled by our definition, and 308 of its 500 items offer a single tool, so the option count alone predicts the label. The number without it is on the multiple-tool half. Without bfcl-live-irrelevance (500 items): AutoJev-27B 98.0%, Kev-9B 97.4%, Shisa DE-1 97.4%.
• Complaint routing. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 13.7%; Bag of words (TF-IDF) 66.8%.
• Commit type. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 19.8%; Bag of words (TF-IDF) 60.6%; Changed paths only (TF-IDF) 54.4%.
• Search relevance. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 29.2%; Bag of words (TF-IDF) 30.3%; Product text kind only 27.0%.

Datasets and provenance

Tasks from public datasets, each with its licence. Two tasks are scored without a source licensed for non‑commercial use, and two internal test sets are not shown.

• JailbreaksOverTime: Piet et al. 2025, CC-BY-4.0.
• Nemotron-PII: NVIDIA, CC-BY-4.0 (attribution requested when results are published).
• Aegis AI Content Safety 2.0: NVIDIA, CC-BY-4.0.
• Banking77: Casanueva et al. 2020, CC-BY-4.0.
• CLINC150: Larson et al. 2019, CC-BY-3.0. MASSIVE: Amazon, CC-BY-4.0. SNIPS: Sonos, CC0-1.0.
• Mind2Web: Deng et al. 2023, CC-BY-4.0 (JevForge snapshot).
• Shopping Queries Dataset (ESCI): Reddy et al. 2022, Apache-2.0.
• Berkeley Function Calling Leaderboard v3 live: Apache-2.0.
• CFPB consumer complaint narratives: US government public domain (BEE-spoke copy, CC0-1.0).
• HaluEval qa and summarization: Li et al. 2023 (HotpotQA CC-BY-SA-4.0, CNN/DailyMail).
• Banking77: Casanueva et al. 2020, CC-BY-4.0.
• Community sets, via Praveenrajus/jev-bench: LEDGAR (Tuggener et al. 2020, CC-BY-4.0), MASSIVE (Amazon, CC-BY-4.0), Civil Comments (Jigsaw, CC0-1.0), Measuring Hate Speech (UC Berkeley D-Lab, CC-BY-4.0), GoEmotions (Google, Apache-2.0), HelpSteer2 helpfulness and verbosity (NVIDIA, CC-BY-4.0), FEVER gold evidence (Thorne et al. 2018, CC-BY-SA-3.0).
• Typed decisions: LocalLLaMA/typed-decisions, Apache-2.0.

Footnotes

1. *Prompt injection, openJev Verdict 1.4: Verdict 1.4 returns a near-constant probability here: the middle 90% of its answers span 0.12 around 0.5, and on off-topic it returned one identical value for all 1,000 items, so this AUROC ranks numerical noise rather than measuring the model. Reading it upside down would not give a real score either. Scoring polarity was checked against the author's own reference adapter and is correct (harness investigation, 2026-09-23).
2. *Every task, GLiNER2.5-Decide: GLiNER2.5-Decide ran behind our own shim over the author's gliner2 2.0.0 package, in FP16 on a 4 GB laptop GPU; the checkpoint is FP32, which spills out of GPU memory on long inputs on that card. Against FP32 on CPU, FP16 picked the same answer on 20 of 20 short inputs (max probability difference 0.0002) and on 54 of 54 long ones of 1,962 to 4,053 tokens (max difference 0.0017). Nothing is truncated (gliner2's default), so long inputs run past the 512 positions of the DeBERTa encoder; label names and descriptions count toward that length, and the author's guidance for long documents is chunking, which this run does not do. Requests over 512 encoded tokens: routing-77 100%, routing-20 82%, complaint-routing 81%, x-typed-decisions 75%, tool-routing 58%, x-jevbench-community 29%, rag-faithfulness 22%, prompt-injection 16%, moderation 15%, x-mind2web-actions 14%, search-relevance 8%, pii 7%, commit-type 6%, off-topic 0.2%. Even in FP16, requests above about 2,300 tokens spill out of GPU memory: 2 prompt-injection requests (73% of that task's measured time) and 1 tool-routing request (4%), so those latencies describe our card more than the model. Its latency is laptop-tier and not comparable with the cloud rows. There is no independent reference to reproduce these numbers against: Fastino's own test split is private.
3. *Prompt injection, GLiNER2.5-Decide: gliner2.5-decide: Related specialty (the card names moderation; the release blog demonstrates prompt injection); no jailbreak dataset is named; no overlap with Fastino's published sample (the training set itself is unpublished).
4. *Prompt injection, Laya-typed: laya-typed: Trained on unnamed jailbreak and prompt-injection data; deepset/prompt-injections is declared held out.
5. *Prompt injection, openJev Verdict 1.4: verdict-1.4: fitted its calibrator on deepset/prompt-injections train split (`use: calibration`); fitted its calibrator on TrustAIRLab in-the-wild jailbreaks (inside djapp18/JailbreaksOverTime) (`use: calibration`): that is 2 of this suite's 4 sources, 50% of its items. Shown for reference, not ranked here.
6. *Prompt injection, Laya: laya: Trained on unnamed jailbreak and prompt-injection data; deepset/prompt-injections is declared held out.
7. *Moderation, AutoJev-27B: autojev-27b: trained on nvidia/Aegis-AI-Content-Safety-Dataset-2.0 test split (`use: weights`): that is 1 of this suite's 3 sources, 33% of its items. Shown for reference, not ranked here. Declared evidence: data.py TRAIN_QUOTAS.
8. *Moderation, GLiNER2.5-Decide: gliner2.5-decide: The model card names moderation as a specialty of its unpublished synthetic training data; none of our sources is named; no overlap with Fastino's published sample (the training set itself is unpublished).
9. *Moderation, Laya-typed: laya-typed: Trained on unnamed human-labelled toxicity data, a closely related task.
10. *Moderation, Laya: laya: Trained on unnamed human-labelled toxicity data, a closely related task.
11. *Moderation, openJev Verdict 1.4: verdict-1.4: Its base model's training lineage includes toxicity and hate-speech datasets, a closely related task.
12. *PII, SimpleJev (Qwen3.5-0.8B): This zero-shot readout saturates at its lowest rating bin (every probability between 0.0102 and 0.0143) and answers 'no PII' to all 1,000 documents. The residual ordering runs backwards because our PII negatives are redacted documents whose stand-in phrases name the very categories the questions ask about ('the address on file', 'the customer'), which is a documented property of the dataset. The score is a dataset artefact, not an inverted signal (harness investigation, 2026-09-23).
13. *PII, GLiNER2.5-Decide: gliner2.5-decide: Fastino's evaluation suite, made by the same generator as the training data, has a contains-PII question; none of our PII sources is named; no overlap with Fastino's published sample (the training set itself is unpublished).
14. *RAG faithfulness, Laya, Laya-typed: Laya does carry signal on one of this framing's three sub-questions (contradicts: AUROC 61 / 64), but that question also returns the lowest probabilities, so the pre-registered max-combine rule almost always hands the item's score to one of the two uninformative sub-questions instead. The result is a combined score at or slightly below chance for both Laya and Laya-typed. This is a limitation of the max-combine framing on this model, not a polarity error; the single-question 'broad' framing is also at chance for Laya here (harness investigation, 2026-09-23).
15. *RAG faithfulness, GLiNER2.5-Decide: GLiNER2.5-Decide's scores are marked degenerate here: its probabilities barely vary (the central 90% spans 0.141 on this subset, just under the 0.15 bar; 0.152 on the full test set, just over it). Its balanced accuracy is 57.8%, and its AUROC of 59.3% [55.1, 63.5] shows the compressed scores order the items only slightly better than chance, so read both numbers as fragile.
16. *RAG faithfulness, GLiNER2.5-Decide: gliner2.5-decide: The model card names yes/no questions over a passage as a capability of its unpublished synthetic training; no NLI or faithfulness dataset is named; no overlap with Fastino's published sample (the training set itself is unpublished).
17. *RAG faithfulness, Kev-0.8B: kev-0.8b: Trained on MNLI and BoolQ, closely related entailment and passage-QA tasks (no item overlap found).
18. *RAG faithfulness, Kev-0.6B: kev-0.6b: Trained on MNLI and BoolQ, closely related entailment and passage-QA tasks (no item overlap found).
19. *RAG faithfulness, openJev Verdict 1.4: verdict-1.4: Its base model's training lineage includes natural-language-inference datasets, a closely related task.
20. *RAG faithfulness, Laya: laya: Trained on NLI and fact-verification data (unnamed), closely related to faithfulness checking.
21. *RAG faithfulness, Laya-typed: laya-typed: Trained on NLI and fact-verification data (unnamed), closely related to faithfulness checking.
22. *Off-topic, GLiNER2.5-Decide: GLiNER2.5-Decide's balanced accuracy here is 49.3%, chance level, and its scores are marked degenerate: its probabilities barely vary (the central 90% spans 0.13). What ordering they have runs the wrong way (AUROC 35.9%): the model gives higher P(yes) to in-scope messages when asked whether a message is outside the listed scope. With a single question its dev AUROC is 0.29 (broad) and 0.26 (detailed); in the chosen decomposed framing the 'unsupported task' question is inverted the same way (test AUROC 0.29) and supplies the max-combined score on 552 of 1,000 items, while 'other subject' alone is correctly oriented (0.71). Scoring polarity is correct: the same yes/no mapping is correctly oriented on every other binary task. This is how the model reads a negated scope question (harness investigation, 2026-09-25).
23. *Off-topic, AutoJev-27B: autojev-27b: trained on mteb/amazon_massive_intent test (en-US, de-DE) split (`use: weights`): that is 1 of this suite's 3 sources, 33% of its items. Shown for reference, not ranked here. Declared evidence: data.py TRAIN_QUOTAS.
24. *Off-topic, Eikos-27B: eikos-27b: selected its checkpoint on clinc/clinc_oos (`use: selection`): that is 1 of this suite's 3 sources, 33% of its items. Shown for reference, not ranked here. Declared evidence: NOTICE: evaluation-only, part of the card's 'general battery'.
25. *Off-topic, Eikos-4B: eikos-4b: selected its checkpoint on clinc/clinc_oos (`use: selection`): that is 1 of this suite's 3 sources, 33% of its items. Shown for reference, not ranked here. Declared evidence: NOTICE: evaluation-only, part of the card's 'general battery'.
26. *Off-topic, Laya: laya: Trained on unnamed intent-classification data, the task family of this category's sources.
27. *Off-topic, Laya-typed: laya-typed: Trained on unnamed intent-classification data, the task family of this category's sources.
28. *Off-topic, openJev Verdict 1.4: verdict-1.4: trained on clinc/clinc_oos oos_train (100 out-of-scope queries only) split (`use: weights`); fitted its calibrator on clinc/clinc_oos test split (`use: calibration`): that is 1 of this suite's 3 sources, 33% of its items. Shown for reference, not ranked here. Declared evidence: training code.
29. *Off-topic, GLiNER2.5-Decide: gliner2.5-decide: The model card names intent classification as a specialty of its unpublished synthetic training data; CLINC, MASSIVE and SNIPS are not named as training data (MASSIVE and SNIPS were zero-shot evaluation sets in the GLiNER2 paper); no overlap with Fastino's published sample (the training set itself is unpublished).
30. *Routing, 20 intents and Routing, 77 intents, Kev-0.8B, Kev-0.6B, openJev Verdict 1.4: Option-definition bug: all 13 routing-20 items whose gold intent is get_physical_card are PIN questions, which contradicts our written definition of that option ('the customer asks about getting a physical card'). Jev, which reads only the definition, gets 1 of 13; Kev-0.8B, trained on Banking77, gets 13 of 13. beneficiary_not_allowed has the same problem. A memorised dataset convention shows up as accuracy. With the affected items and the adjudicated label errors removed, routing-20's first tier is a three-way tie of Kev-0.8B, Jev and Kev-0.6B, and Jev joins routing-77's first tier (data-quality audit, 2026-09-22). The audit itself warns that dropping flagged items favours Jev by construction, because the flagged items are the ones Jev confidently missed, so read that re-scored tier as a bound, not a result.
31. *Routing, 20 intents and Routing, 77 intents, Kev-0.8B, Kev-0.6B: 27 routing-20 and 18 routing-77 messages also appear verbatim in Kev's own locked test set.
32. *Routing, 20 intents and Routing, 77 intents, Kev-4B: kev-4b: trained on legacy-datasets/banking77 (`use: weights`); selected its checkpoint on legacy-datasets/banking77 test split (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: inherited from kev-0.8b; unverified for this checkpoint.
33. *Routing, 20 intents and Routing, 77 intents, Kev-9B: kev-9b: trained on legacy-datasets/banking77 (`use: weights`); selected its checkpoint on legacy-datasets/banking77 test split (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: inherited from kev-0.8b; unverified for this checkpoint.
34. *Routing, 20 intents and Routing, 77 intents, Kev-0.8B: kev-0.8b: trained on legacy-datasets/banking77 (`use: weights`); selected its checkpoint on legacy-datasets/banking77 test split (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: Kev's own development split; also the served temperature.
35. *Routing, 20 intents and Routing, 77 intents, Eikos-27B: eikos-27b: selected its checkpoint on legacy-datasets/banking77 (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: NOTICE: evaluation-only, part of the card's 'general battery'.
36. *Routing, 20 intents and Routing, 77 intents, Kev-0.6B: kev-0.6b: trained on legacy-datasets/banking77 (`use: weights`); selected its checkpoint on legacy-datasets/banking77 test split (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: Kev's own development split.
37. *Routing, 20 intents and Routing, 77 intents, Eikos-4B: eikos-4b: selected its checkpoint on legacy-datasets/banking77 (`use: selection`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: NOTICE: evaluation-only, part of the card's 'general battery'.
38. *Routing, 20 intents and Routing, 77 intents, GLiNER2.5-Decide: gliner2.5-decide: The model card names customer and banking intent as a specialty of its unpublished synthetic training data; Banking77 is not named; no overlap with Fastino's published sample (the training set itself is unpublished).
39. *Routing, 20 intents and Routing, 77 intents, openJev Verdict 1.4: verdict-1.4: trained on legacy-datasets/banking77 train split (`use: weights`); fitted its calibrator on legacy-datasets/banking77 test split (`use: calibration`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: training code.
40. *Routing, 20 intents and Routing, 77 intents, Laya-typed: laya-typed: Trained on unnamed banking-intent data; its authors list Banking77 itself as held out.
41. *Routing, 20 intents and Routing, 77 intents, Laya: laya: Trained on unnamed banking-intent data; its authors list Banking77 itself as held out.
42. *Tool routing, GLiNER2.5-Decide: GLiNER2.5-Decide never picks none_of_these (0 of 1,000 items), so it scores 0% on the bfcl-live-irrelevance half and 64.8% on bfcl-live-multiple; its overall 32.4% is below always-majority (50%) and every trivial baseline.
43. *Complaint routing, GLiNER2.5-Decide: gliner2.5-decide: The model card names email and ticket routing as a specialty of its unpublished synthetic training data; CFPB complaints are not named; no overlap with Fastino's published sample (the training set itself is unpublished).
44. *Complaint routing, Laya-typed: laya-typed: Trained on support-ticket queue routing (Tobi-Bueck/customer-support-tickets), a closely related task.
45. *Complaint routing, Laya: laya: Trained on support-ticket queue routing (Tobi-Bueck/customer-support-tickets), a closely related task.
46. *Search relevance, Laya: laya: Trained on MS MARCO query-passage relevance, a closely related task.
47. *Search relevance, Laya-typed: laya-typed: Trained on MS MARCO query-passage relevance, a closely related task.
48. *Typed decisions, Laya-typed: laya-typed: trained on LocalLLaMA/typed-decisions train split (`use: weights`): that is the whole of this suite. Shown for reference, not ranked here. Declared evidence: model card.
49. *Typed decisions, GLiNER2.5-Decide: gliner2.5-decide: The model card names typed decisions of the same kind (agent completion, severity, routing) as specialties of its unpublished synthetic training; typed-decisions is not named; no overlap with Fastino's published sample (the training set itself is unpublished).
50. *Web-agent actions, GLiNER2.5-Decide: GLiNER2.5-Decide answers this suite's yes/no questions 'yes' 70.5% of the time against 22.6% in the gold labels, so its yes/no accuracy (42.3%) is below always answering 'no' (77.4%).
51. *Community sets, GLiNER2.5-Decide: GLiNER2.5-Decide answers this suite's yes/no questions 'yes' 3% of the time against 50% in the gold labels (yes/no accuracy 49.0%), and 4 of its 8 configs are at or below random: civil_comments 48.7 and fever_evidence 49.3 (random 50), helpsteer2 helpfulness 18.7 and verbosity 16.0 (random 20).
52. *Community sets, AutoJev-27B: autojev-27b: trained on amazon_massive_intent (inside Praveenrajus/jev-bench) test (en-US, de-DE) split (`use: weights`); trained on google/civil_comments (inside Praveenrajus/jev-bench) (`use: weights`): that is 2 of this suite's 8 sources, 25% of its items. Still ranked: the overlap is below the one-third bar. Declared evidence: data.py TRAIN_QUOTAS.
53. *Community sets, Laya-typed: laya-typed: Trained on unnamed toxicity, rubric-rating and fact-checking data whose descriptions match four of this suite's eight configs; possibly the same datasets.
54. *Community sets, Kev-0.8B: kev-0.8b: Trained on MNLI, a task close to this suite's fact-checking config (150 of 1,200 items).
55. *Community sets, Laya: laya: Trained on unnamed toxicity, rubric-rating and fact-checking data whose descriptions match four of this suite's eight configs; possibly the same datasets.
56. *Community sets, Kev-0.6B: kev-0.6b: Trained on MNLI, a task close to this suite's fact-checking config (150 of 1,200 items).
57. *Community sets, GLiNER2.5-Decide: gliner2.5-decide: Partial: the model card names moderation, ordinal scores and intents as specialties, related to some configs of this suite; no overlap with Fastino's published sample (the training set itself is unpublished).
58. *Community sets, openJev Verdict 1.4: verdict-1.4: Its base model's lineage includes toxicity, hate-speech and NLI data, close to three of this suite's eight configs.

Provenance

Release 2026-09-25.1, benchmark commit c372cd5. All 235 runs behind these numbers were made from a working tree with uncommitted harness changes, so the exact code of those runs is not reproducible from a commit alone.

29 cloud‑GPU result rows were completed from the GPU box’s own reports after a copy error cut the local records short; on the 17,553 items both copies hold, they agree on every one.

Verified against the model cards and the dataset licences on 2026-09-25. Spot something outdated? Tell us.