Jev beats every tiny open model

1. 1

   Frozen splits

   Each use case has a 60‑item dev split and a 1,000‑item test split (500 for 77‑way routing), frozen with checksums before any model ran.

2. 2

   Wording chosen on dev

   Every model got each task in two or three wordings; the wording and the decision threshold were picked on the dev split, once, then frozen.

3. 3

   Intervals and tiers

   Scores carry a 95% percentile bootstrap interval over items (2,000 resamples). Tiers come from a paired McNemar test on accuracy, or a paired bootstrap of the metric difference on the items every model shared.

4. 4

   Latency and cost

   Open models ran one at a time on a laptop GPU (NVIDIA RTX 3050 Ti, 4 GB), so their latency is laptop tier. Jev’s latency is measured from the same laptop and includes the network. Local cost is $0.35 per GPU‑hour times measured time. Jev has the highest p95 on 10 of 11 real tasks.

5. 5

   Long inputs

   Inputs over each set’s length limit (2,500 to 12,000 characters) were removed for every model alike before sampling. Inputs a model cuts at its own token window stay in the score; the share of cut inputs is shown next to coverage.

6. 6

   Contamination

   Each model and task pair was audited against the model’s published training, selection and calibration data. Grayed rows trained on the test items or on the same dataset; footnoted rows trained on a related task.

Applies to the whole page

• LangWatch's Instant Evals runs on Jev, and LangWatch ran this benchmark.
• Jev's training data is not disclosed, so its contamination cannot be checked.
• Runs were made from a working tree with uncommitted changes to the harness; the exact code is not reproducible from a commit alone.

The contamination rule

Grayed: trained on the test items or on the same dataset for at least a third of the category. Footnoted: trained on a related task. Unknown: training data not disclosed. Grayed entries are shown for reference and never ranked as the winner. Grayed rows are shown for reference, never ranked.

Jev’s training data is not disclosed, and every open model starts from a pretrained base whose corpus is not itemised. An unmarked row means no known contamination, not proof that there is none.

What the data‑quality audit found

A trivial baseline fitted on the test items beats every open model on 9 of 11 tasks that have one. The notes below say what a better set would change.

• Prompt injection. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 31.0%; Bag of words (TF-IDF) 72.6%. SPML is separable by length alone (length-only AUROC 0.998): its benign items are short questions and its injections long persona prompts. Without it, the best open model changes. Without spml (750 items): Jev 92.5%, Laya-typed 67.7%, Kev-0.8B 53.9%.
• Moderation. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 57.6%; Bag of words (TF-IDF) 76.3%.
• PII. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 3.2%; Bag of words (TF-IDF) 77.2%; Six regexes 68.7%. The gretel and nemotron negatives carry a redaction stand-in ('on file', 'the customer') that no positive has, so a bag-of-words classifier separates them perfectly (AUROC 1.000). The number without them is on privy alone, the one source without that artefact, where the gap between Jev and the best open model is a fraction of the headline gap. Without gretel and nemotron (333 items): Jev 80.7%, Kev-0.8B 61.4%, Kev-0.6B 43.4%.
• RAG faithfulness. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 67.8%; Bag of words (TF-IDF) 54.6%; String-overlap heuristic 68.0%. HaluEval-qa is a length and style detector: 92.8% of its hallucinated answers end in a full stop against 1.2% of the correct ones, and a five-feature string-overlap heuristic (AUROC 0.977) beats Jev (0.934) on it. The number without it is on HaluEval summarization alone. Without halueval-dialogue and halueval-qa (333 items): Jev 75.9%, Kev-0.6B 54.9%, Kev-0.8B 53.3%.
• Off-topic. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 50.8%; Bag of words (TF-IDF) 83.5%. SNIPS is separable by keywords (TF-IDF AUROC 0.999): it holds only two out-of-scope intent types. Without snips (667 items): Jev 91.9%, Kev-0.8B 72.4%, SimpleJev (Qwen3.5-0.8B) 56.5%.
• Routing, 20 intents. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 3.1%; Bag of words (TF-IDF) 66.7%.
• Routing, 77 intents. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 4.6%; Bag of words (TF-IDF) 49.2%.
• Tool routing. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 48.3%; Bag of words (TF-IDF) 49.5%; Option count only 49.2%; Lexical matcher (unsupervised) 63.2%; Option count only (none-of-these detection, AUROC) 0.793. BFCL's irrelevance half marks a request irrelevant when a call cannot be made (a required argument is missing), while this task asks which tool should handle it: 20.7% of that source is mislabelled by our definition, and 308 of its 500 items offer a single tool, so the option count alone predicts the label. The number without it is on the multiple-tool half. Without bfcl-live-irrelevance (500 items): Jev 96.8%, Kev-0.8B 83.8%, Kev-0.6B 77.0%.
• Complaint routing. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 13.7%; Bag of words (TF-IDF) 66.8%.
• Commit type. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 19.8%; Bag of words (TF-IDF) 60.6%; Changed paths only (TF-IDF) 54.4%.
• Search relevance. Reference baselines, fitted with 5‑fold cross‑validation on the test items themselves: Length only 29.2%; Bag of words (TF-IDF) 30.3%; Product text kind only 27.0%.

Datasets and provenance

15 tasks from public datasets, each with its licence. Two sources licensed for non‑commercial use are left out of the published numbers, and two internal test sets are not shown.

• JailbreaksOverTime: Piet et al. 2025, CC-BY-4.0.
• Nemotron-PII: NVIDIA, CC-BY-4.0 (attribution requested when results are published).
• Aegis AI Content Safety 2.0: NVIDIA, CC-BY-4.0.
• Banking77: Casanueva et al. 2020, CC-BY-4.0.
• CLINC150: Larson et al. 2019, CC-BY-3.0. MASSIVE: Amazon, CC-BY-4.0. SNIPS: Sonos, CC0-1.0.
• Mind2Web: Deng et al. 2023, CC-BY-4.0 (JevForge snapshot).
• Shopping Queries Dataset (ESCI): Reddy et al. 2022, Apache-2.0.
• Berkeley Function Calling Leaderboard v3 live: Apache-2.0.
• CFPB consumer complaint narratives: US government public domain (BEE-spoke copy, CC0-1.0).
• HaluEval qa and summarization: Li et al. 2023 (HotpotQA CC-BY-SA-4.0, CNN/DailyMail).
• Banking77: Casanueva et al. 2020, CC-BY-4.0.
• Community sets, via Praveenrajus/jev-bench: LEDGAR (Tuggener et al. 2020, CC-BY-4.0), MASSIVE (Amazon, CC-BY-4.0), Civil Comments (Jigsaw, CC0-1.0), Measuring Hate Speech (UC Berkeley D-Lab, CC-BY-4.0), GoEmotions (Google, Apache-2.0), HelpSteer2 helpfulness and verbosity (NVIDIA, CC-BY-4.0), FEVER gold evidence (Thorne et al. 2018, CC-BY-SA-3.0).
• Typed decisions: LocalLLaMA/typed-decisions, Apache-2.0.
• JevBench public items: fstandhartinger/jevbench, MIT.

Footnotes

1. *Prompt injection, openJev Verdict 1.4: Verdict 1.4 returns a near-constant probability here: the middle 90% of its answers span 0.12 around 0.5, and on off-topic it returned one identical value for all 1,000 items, so this AUROC ranks numerical noise rather than measuring the model. Reading it upside down would not give a real score either. Scoring polarity was checked against the author's own reference adapter and is correct (harness investigation, 2026-09-23).
2. *Prompt injection, Laya-typed, Laya: Trained on unnamed jailbreak and prompt-injection data; deepset/prompt-injections is declared held out.
3. *Prompt injection, openJev Verdict 1.4: Its confidence calibrator (not the model weights) was fitted on prompts from two of this category's sources.
4. *Moderation, Laya-typed, Laya: Trained on unnamed human-labelled toxicity data, a closely related task.
5. *Moderation, openJev Verdict 1.4: Its base model's training lineage includes toxicity and hate-speech datasets, a closely related task.
6. *PII, SimpleJev (Qwen3.5-0.8B): This zero-shot readout saturates at its lowest rating bin (every probability between 0.0102 and 0.0143) and answers 'no PII' to all 1,000 documents. The residual ordering runs backwards because our PII negatives are redacted documents whose stand-in phrases name the very categories the questions ask about ('the address on file', 'the customer'), which is a documented property of the dataset. The score is a dataset artefact, not an inverted signal (harness investigation, 2026-09-23).
7. *RAG faithfulness, Laya, Laya-typed: Laya does carry signal on one of this framing's three sub-questions (contradicts: AUROC 61 / 64), but that question also returns the lowest probabilities, so the pre-registered max-combine rule almost always hands the item's score to one of the two uninformative sub-questions instead. The result is a combined score at or slightly below chance for both Laya and Laya-typed. This is a limitation of the max-combine framing on this model, not a polarity error; the single-question 'broad' framing is also at chance for Laya here (harness investigation, 2026-09-23).
8. *RAG faithfulness, Kev-0.8B, Kev-0.6B: Trained on MNLI and BoolQ, closely related entailment and passage-QA tasks (no item overlap found).
9. *RAG faithfulness, openJev Verdict 1.4: Its base model's training lineage includes natural-language-inference datasets, a closely related task.
10. *RAG faithfulness, Laya, Laya-typed: Trained on NLI and fact-verification data (unnamed), closely related to faithfulness checking.
11. *Off-topic, Laya, Laya-typed: Trained on unnamed intent-classification data, the task family of this category's sources.
12. *Off-topic, openJev Verdict 1.4: Trained on CLINC150's out-of-scope queries, the source of a third of this category, and calibrated on CLINC test queries; shown for reference, not ranked.
13. *Routing, 20 intents, Kev-0.8B, Kev-0.6B, openJev Verdict 1.4: Option-definition bug: all 13 routing-20 items whose gold intent is get_physical_card are PIN questions, which contradicts our written definition of that option ('the customer asks about getting a physical card'). Jev, which reads only the definition, gets 1 of 13; Kev-0.8B, trained on Banking77, gets 13 of 13. beneficiary_not_allowed has the same problem. A memorised dataset convention shows up as accuracy. With the affected items and the adjudicated label errors removed, routing-20's first tier is a three-way tie of Kev-0.8B, Jev and Kev-0.6B, and Jev joins routing-77's first tier (data-quality audit, 2026-09-22). The audit itself warns that dropping flagged items favours Jev by construction, because the flagged items are the ones Jev confidently missed, so read that re-scored tier as a bound, not a result.
14. *Routing, 20 intents, Kev-0.8B, Kev-0.6B: 27 routing-20 and 18 routing-77 messages also appear verbatim in Kev's own locked test set.
15. *Routing, 20 intents, Kev-0.8B, Kev-0.6B: Trained on Banking77, the dataset behind this category (train split), and its checkpoint was picked on a set holding 27 of these 1,000 test messages; shown for reference, not ranked.
16. *Routing, 20 intents, openJev Verdict 1.4: Trained on Banking77, the dataset behind this category (train split), and calibrated on Banking77 test messages; shown for reference, not ranked.
17. *Routing, 20 intents, Laya-typed, Laya: Trained on unnamed banking-intent data; its authors list Banking77 itself as held out.
18. *Routing, 77 intents, Kev-0.8B, Kev-0.6B: Trained on Banking77, the dataset behind this category (train split), and its checkpoint was picked on a set holding 16 of these 500 test messages; shown for reference, not ranked.
19. *Complaint routing, Laya-typed, Laya: Trained on support-ticket queue routing (Tobi-Bueck/customer-support-tickets), a closely related task.
20. *Search relevance, Laya, Laya-typed: Trained on MS MARCO query-passage relevance, a closely related task.
21. *Typed decisions, Laya-typed: Fine-tuned on this dataset's training split (same generator, workflows and questions); shown for reference, not ranked.
22. *Community sets, Laya-typed, Laya: Trained on unnamed toxicity, rubric-rating and fact-checking data whose descriptions match four of this suite's eight configs; possibly the same datasets.
23. *Community sets, Kev-0.8B, Kev-0.6B: Trained on MNLI, a task close to this suite's fact-checking config (150 of 1,200 items).
24. *Community sets, openJev Verdict 1.4: Its base model's lineage includes toxicity, hate-speech and NLI data, close to three of this suite's eight configs.
25. *JevBench public, Kev-0.6B, Kev-0.8B: Trained on template-generated policy-rule cases like this set's policy items (no item overlap found).
26. *JevBench public, openJev Verdict 1.4: openJev Verdict 1.4's inference settings were chosen by measuring on these same 231 public items; shown for reference, not ranked.

Provenance

Release 2026-09-23.1, benchmark commit 58b54bf. 120 of the runs behind these numbers were made from a working tree with uncommitted harness changes, so the exact code of those runs is not reproducible from a commit alone.

Jev’s cost per 1,000 decisions is not shown on this release.

Verified against the model cards, the dataset licences and the public list price on 2026-09-22. Spot something outdated? Tell us.