LangWatch
Sign Up

TypeSafeProprietary model

Jev benchmarks

All models
  • General rank
    #2of 27Leads 30 of 44 (68%)non-coding tasks
  • Speed#1 / 7
    ~65 msMedian latency, p50Hosted API only
  • Cost#1 / 8
    $0.029Per 1,000 decisionsHosted price
  • LLM-as-a-judge#1 / 20 tied
    72.5%Best task · Accuracyvs Jev-class models

Comparison summary

Strongest
Agents
Leads 2 of 2 (100%)
Weakest
Judging and evals
Leads 3 of 6 (50%)

Strong at agent actions and general decisions, weaker at judging. It leads 30 of 44 (68%) non-coding tasks, #2 of 27 models by that count.

Specifications

Lab
TypeSafe
Weights
Proprietary
Parameters
Not disclosed
Licence
Proprietary

Overall standing

Share of non-coding tasks led · #2 of 27

  1. Claude Opus 5.585%
  2. Jev
    68%
  3. AutoJev-27B67%
  4. Eikos-27B60%
  5. GPT-6.1 Sol54%
  6. Claude Sonnet 5.546%
  7. Shisa DE-142%
  8. Gemini 3.8 Flash38%

Safety and guardrails

Share of tasks led · #4 of 20

  1. Eikos-27BLeads 3 of 3
  2. AutoJev-27BLeads 2 of 2
  3. Claude Opus 5.5Leads 3 of 4
  4. Jev
    Leads 8 of 12
  5. GPT-6.1 SolLeads 2 of 4
  6. Claude Sonnet 5.5Leads 2 of 4
  7. Shisa DE-1Leads 1 of 4
  8. Gemini 3.8 FlashLeads 1 of 4
See the 4 tasks

Prompt injection

Ahead alone#1 / 8

Balanced accuracy, % · Higher is better

  1. Jev
    95.2, interval 93.9 to 96.5, ahead alone
  2. Claude Haiku 4.592.7, interval 91.1 to 94.3, tier 2
  3. GPT-6.1 Sol91.3, interval 89.6 to 92.9, tier 2
  4. Claude Opus 5.590.9, interval 89.2 to 92.5, tier 2
  5. Gemini 3.8 Flash90.4, interval 88.6 to 92.1, tier 3
  6. Claude Sonnet 5.588.3, interval 86.4 to 90.2, tier 4
  7. GPT-5.6 Terra88.2, interval 86.3 to 90.0, tier 4
  8. GPT-6 Luna88.0, interval 86.1 to 89.8, tier 4
Tested against 7 frontier LLMsAlso tested against open models: #1 of 12 tiedAlso tested against small open models: #1 of 4

Moderation

Tier 2 of 2#8 / 8

Balanced accuracy, % · Higher is better

  1. Claude Sonnet 5.581.6, interval 78.5 to 84.4, tied for the lead
  2. GPT-6.1 Sol81.4, interval 78.5 to 84.2, tied for the lead
  3. Gemini 3.8 Flash81.4, interval 78.4 to 84.1, tied for the lead
  4. Claude Opus 5.581.1, interval 78.0 to 84.0, tied for the lead
  5. Claude Haiku 4.581.1, interval 78.0 to 84.0, tied for the lead
  6. GPT-6 Luna80.8, interval 77.6 to 83.7, tied for the lead
  7. GPT-5.6 Terra79.9, interval 77.0 to 82.9, tied for the lead
  8. Jev
    78.0, interval 74.9 to 81.0, tier 2
Tested against 7 frontier LLMsAlso tested against open models: #1 of 12 tiedAlso tested against small open models: #1 of 5

Off-topic

Tier 3 of 4#5 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.596.0, interval 94.8 to 97.1, tied for the lead
  2. Claude Sonnet 5.595.4, interval 94.1 to 96.6, tied for the lead
  3. GPT-6.1 Sol95.3, interval 93.9 to 96.6, tied for the lead
  4. Gemini 3.8 Flash95.0, interval 93.6 to 96.3, tier 2
  5. Jev
    93.4, interval 91.8 to 94.9, tier 3
  6. GPT-5.6 Terra92.3, interval 90.6 to 93.9, tier 3
  7. Claude Haiku 4.591.5, interval 89.7 to 93.2, tier 4
  8. GPT-6 Luna90.6, interval 88.7 to 92.4, tier 4
Tested against 7 frontier LLMsAlso tested against open models: #1 of 9Also tested against small open models: #1 of 5

PII

Tier 2 of 5#2 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.595.4, interval 94.1 to 96.6, ahead alone
  2. Claude Sonnet 5.594.0, interval 92.6 to 95.5, tier 2
  3. Gemini 3.8 Flash93.1, interval 91.5 to 94.6, tier 2
  4. Jev
    92.9, interval 91.3 to 94.4, tier 2
  5. GPT-6.1 Sol91.3, interval 89.7 to 93.0, tier 3
  6. GPT-6 Luna87.2, interval 85.3 to 89.1, tier 4
  7. GPT-5.6 Terra86.4, interval 84.5 to 88.4, tier 4
  8. Claude Haiku 4.583.1, interval 80.9 to 85.3, tier 5
Tested against 7 frontier LLMsAlso tested against open models: #3 of 13Also tested against small open models: #1 of 5

Routing and classification

Share of tasks led · #4 of 23

  1. AutoJev-27BLeads 4 of 5
  2. Shisa DE-1Leads 3 of 4
  3. Claude Opus 5.5Leads 3 of 4
  4. Jev
    Leads 9 of 14
  5. GPT-5.6 TerraLeads 2 of 4
  6. Eikos-27BLeads 1 of 3
  7. SemIf (Qwen3.5-4B)Leads 1 of 3
  8. Gemini 3.8 FlashLeads 1 of 4
See the 5 tasks

Routing, 20 intents

Tier 3 of 3#6 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.595.0, interval 93.6 to 96.3, ahead alone
  2. GPT-6.1 Sol92.8, interval 91.2 to 94.3, tier 2
  3. Gemini 3.8 Flash92.7, interval 91.1 to 94.2, tier 2
  4. GPT-5.6 Terra92.4, interval 90.7 to 94.0, tier 2
  5. Claude Sonnet 5.591.7, interval 89.9 to 93.4, tier 2
  6. GPT-6 Luna90.3, interval 88.4 to 92.1, tier 3
  7. Claude Haiku 4.589.3, interval 87.3 to 91.2, tier 3
  8. Jev
    89.1, interval 87.0 to 91.0, tier 3
Tested against 7 frontier LLMsAlso tested against open models: #2 of 7 tiedAlso tested against small open models: #1 of 4

Routing, 77 intents

Tier 4 of 4#8 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.588.0, interval 85.2 to 90.8, ahead alone
  2. Gemini 3.8 Flash84.8, interval 81.6 to 88.0, tier 2
  3. GPT-6.1 Sol84.4, interval 81.2 to 87.6, tier 2
  4. Claude Sonnet 5.584.2, interval 81.0 to 87.2, tier 2
  5. GPT-5.6 Terra83.4, interval 80.0 to 86.6, tier 2
  6. GPT-6 Luna82.2, interval 78.8 to 85.6, tier 3
  7. Claude Haiku 4.580.0, interval 76.6 to 83.4, tier 3
  8. Jev
    79.6, interval 76.0 to 83.0, tier 4
Tested against 7 frontier LLMsAlso tested against open models: #1 of 5 tiedAlso tested against small open models: #1 of 3

Tool routing

Tier 4 of 4#7 / 8

Accuracy, % · Higher is better

  1. GPT-6 Luna85.5, interval 83.3 to 87.6, tied for the lead
  2. GPT-5.6 Terra84.2, interval 82.0 to 86.3, tied for the lead
  3. Gemini 3.8 Flash83.6, interval 81.3 to 85.7, tier 2
  4. Claude Haiku 4.582.0, interval 79.6 to 84.2, tier 2
  5. GPT-6.1 Sol79.9, interval 77.4 to 82.2, tier 3
  6. Claude Sonnet 5.579.4, interval 76.9 to 81.7, tier 3
  7. Claude Opus 5.578.7, interval 76.1 to 81.1, tier 4
  8. Jev
    78.3, interval 75.7 to 80.7, tier 4
Tested against 7 frontier LLMsAlso tested against open models: #4 of 16 tiedAlso tested against small open models: #1 of 8

Complaint routing

Tied for the lead#1 / 8

Accuracy, % · Higher is better

  1. Gemini 3.8 Flash79.6, interval 77.1 to 82.0, tied for the lead
  2. GPT-6.1 Sol79.0, interval 76.5 to 81.4, tied for the lead
  3. Claude Opus 5.578.9, interval 76.4 to 81.4, tied for the lead
  4. Jev
    78.7, interval 76.3 to 81.2, tied for the lead
  5. GPT-5.6 Terra78.4, interval 75.9 to 81.0, tied for the lead
  6. Claude Sonnet 5.578.3, interval 75.7 to 80.8, tied for the lead
  7. GPT-6 Luna77.9, interval 75.4 to 80.4, tier 2
  8. Claude Haiku 4.577.1, interval 74.6 to 79.7, tier 2
Tested against 7 frontier LLMsAlso tested against open models: #1 of 15 tiedAlso tested against small open models: #1 of 8

Typed decisions

Tied for the lead#1 / 14

Accuracy, % · Higher is better

  1. Laya-typed*77.4*, interval 75.4 to 79.4, trained on this data, not ranked
  2. Shisa DE-175.1, interval 72.8 to 77.3, tied for the lead
  3. Jev
    73.9, interval 71.8 to 76.0, tied for the lead
  4. AutoJev-27B73.8, interval 71.7 to 75.9, tied for the lead
  5. Eikos-27B73.4, interval 71.2 to 75.6, tied for the lead
  6. Kev-4B65.8, interval 63.6 to 68.0, tier 2
  7. Eikos-4B65.3, interval 62.7 to 68.0, tier 2
  8. SemIf (Qwen3.5-4B)63.0, interval 60.5 to 65.4, tier 3
Tested against 14 open modelsAlso tested against small open models: #1 of 7

Grounding

Share of tasks led · #6 of 22

  1. Claude Opus 5.5Leads 2 of 2
  2. Claude Sonnet 5.5Leads 2 of 2
  3. Eikos-27BLeads 2 of 2
  4. Gemini 3.8 FlashLeads 2 of 2
  5. GPT-6.1 SolLeads 2 of 2
  6. Jev
    Leads 4 of 6
  7. AutoJev-27BLeads 0 of 2
  8. Shisa DE-1Leads 0 of 2
See the 2 tasks

RAG faithfulness

Tier 2 of 3#5 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.585.0, interval 82.3 to 87.5, tied for the lead
  2. GPT-6.1 Sol84.2, interval 81.5 to 86.9, tied for the lead
  3. Claude Sonnet 5.583.6, interval 80.8 to 86.4, tied for the lead
  4. Gemini 3.8 Flash83.3, interval 80.4 to 86.2, tied for the lead
  5. GPT-5.6 Terra80.6, interval 77.7 to 83.6, tier 2
  6. Jev
    80.3, interval 77.5 to 83.1, tier 2
  7. GPT-6 Luna79.7, interval 76.7 to 82.7, tier 2
  8. Claude Haiku 4.577.4, interval 74.4 to 80.3, tier 3
Tested against 7 frontier LLMsAlso tested against open models: #1 of 13 tiedAlso tested against small open models: #1 of 6

Search relevance

Tier 2 of 3#5 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.563.6, interval 60.7 to 66.7, tied for the lead
  2. GPT-6.1 Sol63.0, interval 60.0 to 66.1, tied for the lead
  3. Gemini 3.8 Flash62.5, interval 59.6 to 65.6, tied for the lead
  4. Claude Sonnet 5.561.1, interval 58.1 to 64.0, tied for the lead
  5. Jev
    57.7, interval 54.7 to 60.8, tier 2
  6. GPT-5.6 Terra57.6, interval 54.3 to 60.6, tier 2
  7. GPT-6 Luna54.7, interval 51.6 to 57.8, tier 2
  8. Claude Haiku 4.550.6, interval 47.4 to 53.7, tier 3
Tested against 7 frontier LLMsAlso tested against open models: #1 of 16 tiedAlso tested against small open models: #1 of 8

Agents

Share of tasks led · #1 of 8

  1. Jev
    Leads 2 of 2
  2. SimpleJev (Qwen3.5-0.8B)Leads 0 of 2
  3. Kev-0.8BLeads 0 of 2
  4. Kev-0.6BLeads 0 of 2
  5. LayaLeads 0 of 2
  6. Laya-typedLeads 0 of 2
  7. openJev Verdict 1.4Leads 0 of 2
  8. SemIf (Qwen3-0.6B)Leads 0 of 2
See the task

Web-agent actions

Ahead alone#1 / 15

Accuracy, % · Higher is better

  1. Jev
    70.8, interval 68.9 to 72.6, ahead alone
  2. Eikos-27B68.4, interval 66.5 to 70.4, tier 2
  3. AutoJev-27B68.2, interval 66.2 to 70.2, tier 2
  4. Shisa DE-165.2, interval 63.3 to 67.2, tier 3
  5. Eikos-4B62.9, interval 60.9 to 64.8, tier 4
  6. SemIf (Qwen3.5-4B)59.5, interval 57.5 to 61.5, tier 5
  7. Kev-4B58.0, interval 55.9 to 60.2, tier 5
  8. SimpleJev (Qwen3.5-0.8B)56.3, interval 54.7 to 57.8, tier 6
Tested against 14 open modelsAlso tested against small open models: #1 of 8

General decisions

Share of tasks led · #1 of 10

  1. Jev
    Leads 4 of 4
  2. AutoJev-27BLeads 2 of 2
  3. Kev-0.6BLeads 0 of 4
  4. Kev-0.8BLeads 0 of 4
  5. LayaLeads 0 of 4
  6. Laya-typedLeads 0 of 4
  7. SimpleJev (Qwen3.5-0.8B)Leads 0 of 2
  8. Kev-4BLeads 0 of 2
See the 2 tasks

Community sets

Tied for the lead#1 / 10

Accuracy, % · Higher is better

  1. AutoJev-27B63.0, interval 60.2 to 65.8, tied for the lead
  2. Jev
    62.3, interval 59.5 to 65.1, tied for the lead
  3. Eikos-27B59.8, interval 56.9 to 62.6, tier 2
  4. Eikos-4B55.8, interval 52.9 to 58.7, tier 3
  5. Kev-4B55.8, interval 53.0 to 58.7, tier 3
  6. Laya-typed47.8, interval 44.9 to 50.5, tier 4
  7. Kev-0.8B46.4, interval 43.5 to 49.3, tier 4
  8. Laya45.9, interval 43.1 to 48.8, tier 4
Tested against 14 open modelsAlso tested against small open models: #1 of 5

JevBench public

Tied for the lead#1 / 12

Accuracy, % · Higher is better

  1. Eikos-27B*91.8*, interval 87.9 to 95.2, trained on this data, not ranked
  2. AutoJev-27B86.6, interval 81.9 to 90.9, tied for the lead
  3. Shisa DE-186.1, interval 81.2 to 90.6, tied for the lead
  4. Jev
    85.7, interval 81.0 to 90.1, tied for the lead
  5. Eikos-4B*85.3*, interval 80.4 to 89.8, trained on this data, not ranked
  6. SemIf (Qwen3.5-4B)80.1, interval 74.7 to 85.3, tier 2
  7. Kev-4B71.4, interval 65.2 to 77.4, tier 3
  8. Kev-0.6B60.6, interval 53.8 to 67.1, tier 4
Tested against 14 open modelsAlso tested against small open models: #1 of 7

Judging and evals

Set

Share of tasks led · #9 of 24

  1. Claude Opus 5.5Leads 3 of 3
  2. ClefLeads 3 of 3
  3. Decision 2.0 Vega 27BLeads 3 of 3
  4. GLiDELeads 3 of 3
  5. Jev-OmniLeads 3 of 3
  6. Kev-27BLeads 3 of 3
  7. WinnowLeads 3 of 3
  8. Jev
    Leads 3 of 6
See the 3 tasks

LLM-as-a-judge

Tier 3 of 3#5 / 5

Accuracy, % · Higher is better

  1. GPT-6.1 Sol92.6, interval 89.1 to 95.8, tied for the lead
  2. Claude Opus 5.589.1, interval 83.9 to 93.9, tied for the lead
  3. Claude Sonnet 5.583.5, interval 78.4 to 88.5, tier 2
  4. Gemini 3.8 Flash80.9, interval 76.2 to 85.5, tier 2
  5. Jev
    61.3, interval 54.8 to 67.9, tier 3
Tested against 4 frontier LLMsAlso tested against Jev-class models: #1 of 20 tied

Scenario judge

Tier 3 of 3#5 / 5

Accuracy, % · Higher is better

  1. Claude Opus 5.586.8, interval 80.2 to 92.3, tied for the lead
  2. Gemini 3.8 Flash83.3, interval 76.8 to 89.1, tied for the lead
  3. Claude Sonnet 5.581.3, interval 74.1 to 87.4, tier 2
  4. GPT-6.1 Sol81.3, interval 74.1 to 87.3, tier 2
  5. Jev
    66.7, interval 57.3 to 75.3, tier 3
Tested against 4 frontier LLMsAlso tested against Jev-class models: #1 of 20 tied

Search

Tier 3 of 3#5 / 5

Accuracy, % · Higher is better

  1. Claude Opus 5.591.5, interval 87.6 to 94.8, tied for the lead
  2. GPT-6.1 Sol89.6, interval 85.6 to 93.1, tied for the lead
  3. Claude Sonnet 5.589.1, interval 85.8 to 92.1, tied for the lead
  4. Gemini 3.8 Flash85.0, interval 81.3 to 88.4, tier 2
  5. Jev
    63.3, interval 58.2 to 68.1, tier 3
Tested against 4 frontier LLMsAlso tested against Jev-class models: #1 of 20 tied

Speed

Median latency, p50 · Hosted API only · lower is better

  1. Jev
    ~65 ms
  2. GPT-6 Luna1,124 ms
  3. Claude Haiku 4.51,306 ms
  4. GPT-5.6 Terra1,565 ms
  5. Claude Sonnet 5.51,646 ms
  6. Gemini 3.8 Flash1,936 ms
  7. Claude Opus 5.52,194 ms

Cost

Per 1,000 decisions, hosted price · lower is better

  1. Jev
    $0.029
  2. GPT-6 Luna$0.049
  3. Gemini 3.8 Flash$0.57
  4. Claude Haiku 4.5$0.69
  5. GPT-6.1 Sol~$1.02
  6. GPT-5.6 Terra$1.03
  7. Claude Sonnet 5.5$1.52
  8. Claude Opus 5.5$3.44
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

JevOther modelsTrained on the test data*Whisker: 95% interval