LangWatch
Sign Up

shisa-aiOpen weights model

Shisa DE-1 benchmarks

All models
  • General rank
    #7of 27Leads 5 of 12 (42%)non-coding tasks
  • Speed#2 / 4
    46 msMedian latency, p50H100 only
  • Cost
    Self-hostedRan on H100No hosted price
  • Tool routing#1 / 16 tied
    81.3%Best task · Accuracyvs open models

Comparison summary

Strongest
General decisions
Leads 1 of 1 (100%)
Weakest
Agents
Leads 0 of 1 (0%)

Strong at general decisions and routing, weaker at agent actions. It leads 5 of 12 (42%) non-coding tasks, #7 of 27 models by that count.

Specifications

Lab
shisa-ai
Weights
Open weights
Parameters
Not disclosed
Licence
Not recorded

Overall standing

Share of non-coding tasks led · #7 of 27

  1. Claude Opus 5.585%
  2. Jev68%
  3. AutoJev-27B67%
  4. Eikos-27B60%
  5. GPT-6.1 Sol54%
  6. Claude Sonnet 5.546%
  7. Shisa DE-1
    42%
  8. Gemini 3.8 Flash38%

Safety and guardrails

Share of tasks led · #4 of 13

  1. Eikos-27BLeads 3 of 3
  2. AutoJev-27BLeads 2 of 2
  3. JevLeads 8 of 12
  4. Shisa DE-1
    Leads 1 of 4
  5. Kev-9BLeads 0 of 4
  6. Eikos-4BLeads 0 of 3
  7. SemIf (Qwen3.5-4B)Leads 0 of 4
  8. Kev-0.8BLeads 0 of 8
See the 4 tasks

Prompt injection

Tied for the lead#1 / 12

Catch rate at 5% false alarms, % · Higher is better

  1. Jev94.6, interval 92.2 to 96.7, tied for the lead
  2. Eikos-27B94.2, interval 90.5 to 97.1, tied for the lead
  3. Shisa DE-1
    94.0, interval 90.3 to 96.2, tied for the lead
  4. AutoJev-27B92.0, interval 86.6 to 95.0, tied for the lead
  5. Kev-4B91.6, interval 88.2 to 94.2, tier 2
  6. Eikos-4B90.8, interval 88.3 to 94.0, tier 2
  7. SemIf (Qwen3.5-4B)90.6, interval 87.8 to 93.5, tier 2
  8. Kev-9B87.6, interval 82.4 to 92.5, tier 2
Tested against Jev and 14 open models

Moderation

Tier 2 of 6#3 / 12

AUROC, % · Higher is better

  1. AutoJev-27B*90.6*, interval 88.4 to 92.7, trained on this data, not ranked
  2. Eikos-27B90.3, interval 87.9 to 92.4, tied for the lead
  3. Jev90.3, interval 87.9 to 92.4, tied for the lead
  4. Eikos-4B88.8, interval 86.3 to 91.3, tier 2
  5. SemIf (Qwen3.5-4B)88.7, interval 86.2 to 91.2, tier 2
  6. Kev-9B88.5, interval 86.0 to 90.9, tier 2
  7. Shisa DE-1
    88.3, interval 85.6 to 90.8, tier 2
  8. Kev-4B82.0, interval 78.8 to 85.0, tier 3
Tested against Jev and 14 open models

Off-topic

Tier 3 of 8#3 / 9

Balanced accuracy, % · Higher is better

  1. Jev93.4, interval 91.8 to 94.9, ahead alone
  2. AutoJev-27B*92.7*, interval 91.0 to 94.4, trained on this data, not ranked
  3. Eikos-27B*90.6*, interval 88.8 to 92.4, trained on this data, not ranked
  4. Kev-9B89.6, interval 87.7 to 91.5, tier 2
  5. Shisa DE-1
    88.0, interval 86.1 to 89.9, tier 3
  6. Kev-4B84.4, interval 82.1 to 86.7, tier 4
  7. SemIf (Qwen3.5-4B)81.1, interval 79.0 to 83.4, tier 5
  8. Eikos-4B*79.6*, interval 77.1 to 82.0, trained on this data, not ranked
Tested against Jev and 14 open models

PII

Tier 3 of 5#4 / 13

Catch rate at 5% false alarms, % · Higher is better

  1. Eikos-27B95.2, interval 92.8 to 97.8, tied for the lead
  2. AutoJev-27B92.8, interval 90.2 to 95.5, tied for the lead
  3. Jev90.8, interval 88.1 to 93.8, tier 2
  4. Shisa DE-1
    84.2, interval 77.3 to 89.1, tier 3
  5. Eikos-4B78.6, interval 74.5 to 85.3, tier 3
  6. SemIf (Qwen3.5-4B)66.1, interval 60.9 to 74.6, tier 4
  7. Kev-9B62.5, interval 0.2 to 66.4, tier 4
  8. Kev-0.8B22.8, interval 6.1 to 26.3, tier 5
Tested against Jev and 14 open models

Routing and classification

Share of tasks led · #2 of 16

  1. AutoJev-27BLeads 4 of 5
  2. Shisa DE-1
    Leads 3 of 4
  3. JevLeads 9 of 14
  4. Eikos-27BLeads 1 of 3
  5. SemIf (Qwen3.5-4B)Leads 1 of 3
  6. Kev-4BLeads 0 of 3
  7. Kev-9BLeads 0 of 2
  8. Eikos-4BLeads 0 of 2
See the 5 tasks

Routing, 20 intents

Tier 2 of 5#2 / 7

Accuracy, % · Higher is better

  1. Kev-4B*93.0*, interval 91.4 to 94.5, trained on this data, not ranked
  2. Kev-9B*93.0*, interval 91.4 to 94.6, trained on this data, not ranked
  3. Kev-0.8B*91.3*, interval 89.5 to 93.0, trained on this data, not ranked
  4. AutoJev-27B89.7, interval 87.7 to 91.6, ahead alone
  5. Eikos-27B*89.6*, interval 87.7 to 91.4, trained on this data, not ranked
  6. Kev-0.6B*89.5*, interval 87.5 to 91.5, trained on this data, not ranked
  7. Jev89.1, interval 87.0 to 91.0, tier 2
  8. Shisa DE-1
    88.1, interval 85.9 to 90.1, tier 2
Tested against Jev and 14 open models

Routing, 77 intents

Not ranked

Accuracy, % · Higher is better

  1. Kev-4B*85.0*, interval 81.8 to 88.0, trained on this data, not ranked
  2. Kev-9B*85.0*, interval 81.8 to 88.2, trained on this data, not ranked
  3. Kev-0.8B*83.0*, interval 79.6 to 86.4, trained on this data, not ranked
  4. AutoJev-27B80.6, interval 77.0 to 84.0, tied for the lead
  5. Jev79.6, interval 76.0 to 83.0, tied for the lead
  6. Kev-0.6B*78.2*, interval 74.4 to 81.8, trained on this data, not ranked
  7. Eikos-27B*77.8*, interval 74.0 to 81.4, trained on this data, not ranked
  8. Shisa DE-1
    0.0, interval 0.0 to 0.0, not ranked
Tested against Jev and 14 open models

Tool routing

Tied for the lead#1 / 16

Accuracy, % · Higher is better

  1. Shisa DE-1
    81.3, interval 78.9 to 83.5, tied for the lead
  2. AutoJev-27B80.6, interval 78.1 to 82.9, tied for the lead
  3. SemIf (Qwen3.5-4B)79.9, interval 77.4 to 82.2, tied for the lead
  4. Eikos-27B79.3, interval 76.8 to 81.8, tier 2
  5. Jev78.3, interval 75.7 to 80.7, tier 2
  6. Eikos-4B75.5, interval 72.8 to 78.0, tier 3
  7. Kev-9B72.4, interval 69.7 to 75.0, tier 4
  8. Kev-4B71.4, interval 68.6 to 74.0, tier 4
Tested against Jev and 14 open models

Complaint routing

Tied for the lead#1 / 15

Accuracy, % · Higher is better

  1. Jev78.7, interval 76.3 to 81.2, tied for the lead
  2. Shisa DE-1
    78.0, interval 75.4 to 80.6, tied for the lead
  3. Kev-4B76.2, interval 73.6 to 79.0, tier 2
  4. AutoJev-27B75.9, interval 73.2 to 78.7, tier 2
  5. Eikos-27B75.6, interval 73.0 to 78.4, tier 2
  6. Kev-9B75.0, interval 72.4 to 77.7, tier 2
  7. SemIf (Qwen3.5-4B)72.9, interval 70.2 to 75.7, tier 3
  8. Kev-0.6B59.3, interval 56.3 to 62.3, tier 4
Tested against Jev and 13 open models

Typed decisions

Tied for the lead#1 / 14

Accuracy, % · Higher is better

  1. Laya-typed*77.4*, interval 75.4 to 79.4, trained on this data, not ranked
  2. Shisa DE-1
    75.1, interval 72.8 to 77.3, tied for the lead
  3. Jev73.9, interval 71.8 to 76.0, tied for the lead
  4. AutoJev-27B73.8, interval 71.7 to 75.9, tied for the lead
  5. Eikos-27B73.4, interval 71.2 to 75.6, tied for the lead
  6. Kev-4B65.8, interval 63.6 to 68.0, tier 2
  7. Eikos-4B65.3, interval 62.7 to 68.0, tier 2
  8. SemIf (Qwen3.5-4B)63.0, interval 60.5 to 65.4, tier 3
Tested against Jev and 13 open models

Grounding

Share of tasks led · #4 of 15

  1. Eikos-27BLeads 2 of 2
  2. JevLeads 4 of 6
  3. AutoJev-27BLeads 0 of 2
  4. Shisa DE-1
    Leads 0 of 2
  5. Eikos-4BLeads 0 of 2
  6. Kev-4BLeads 0 of 2
  7. Kev-9BLeads 0 of 2
  8. Kev-0.8BLeads 0 of 4
See the 2 tasks

RAG faithfulness

Tier 2 of 5#3 / 13

Balanced accuracy, % · Higher is better

  1. Eikos-27B82.1, interval 79.2 to 84.9, tied for the lead
  2. Jev80.3, interval 77.5 to 83.1, tied for the lead
  3. Shisa DE-1
    79.2, interval 76.2 to 82.2, tier 2
  4. AutoJev-27B79.1, interval 76.1 to 82.0, tier 2
  5. Kev-9B73.1, interval 70.0 to 76.0, tier 3
  6. Kev-4B72.5, interval 69.4 to 75.5, tier 3
  7. SemIf (Qwen3.5-4B)72.0, interval 68.9 to 75.2, tier 3
  8. Eikos-4B71.0, interval 67.4 to 74.5, tier 3
Tested against Jev and 14 open models

Search relevance

Tier 2 of 7#3 / 16

Accuracy, % · Higher is better

  1. Jev57.7, interval 54.7 to 60.8, tied for the lead
  2. Eikos-27B56.6, interval 53.4 to 59.7, tied for the lead
  3. AutoJev-27B52.8, interval 49.6 to 55.9, tier 2
  4. Shisa DE-1
    50.3, interval 47.1 to 53.6, tier 2
  5. Kev-4B44.8, interval 41.7 to 48.0, tier 3
  6. Kev-9B43.4, interval 40.2 to 46.5, tier 3
  7. Eikos-4B43.3, interval 40.1 to 46.3, tier 3
  8. SemIf (Qwen3.5-4B)41.7, interval 38.6 to 44.7, tier 4
Tested against Jev and 14 open models

Agents

Leads 0 of 1, too few tasks to compare here.

See the task

Web-agent actions

Tier 3 of 10#4 / 15

Accuracy, % · Higher is better

  1. Jev70.8, interval 68.9 to 72.6, ahead alone
  2. Eikos-27B68.4, interval 66.5 to 70.4, tier 2
  3. AutoJev-27B68.2, interval 66.2 to 70.2, tier 2
  4. Shisa DE-1
    65.2, interval 63.3 to 67.2, tier 3
  5. Eikos-4B62.9, interval 60.9 to 64.8, tier 4
  6. SemIf (Qwen3.5-4B)59.5, interval 57.5 to 61.5, tier 5
  7. Kev-4B58.0, interval 55.9 to 60.2, tier 5
  8. SimpleJev (Qwen3.5-0.8B)56.3, interval 54.7 to 57.8, tier 6
Tested against Jev and 13 open models

General decisions

Leads 1 of 1, too few tasks to compare here.

See the 2 tasks

Community sets

Not ranked

Accuracy, % · Higher is better

  1. AutoJev-27B63.0, interval 60.2 to 65.8, tied for the lead
  2. Jev62.3, interval 59.5 to 65.1, tied for the lead
  3. Eikos-27B59.8, interval 56.9 to 62.6, tier 2
  4. Eikos-4B55.8, interval 52.9 to 58.7, tier 3
  5. Kev-4B55.8, interval 53.0 to 58.7, tier 3
  6. Laya-typed47.8, interval 44.9 to 50.5, tier 4
  7. Kev-0.8B46.4, interval 43.5 to 49.3, tier 4
  8. Shisa DE-1
    38.2, interval 35.4 to 40.9, not ranked
Tested against Jev and 13 open models

JevBench public

Tied for the lead#1 / 12

Accuracy, % · Higher is better

  1. Eikos-27B*91.8*, interval 87.9 to 95.2, trained on this data, not ranked
  2. AutoJev-27B86.6, interval 81.9 to 90.9, tied for the lead
  3. Shisa DE-1
    86.1, interval 81.2 to 90.6, tied for the lead
  4. Jev85.7, interval 81.0 to 90.1, tied for the lead
  5. Eikos-4B*85.3*, interval 80.4 to 89.8, trained on this data, not ranked
  6. SemIf (Qwen3.5-4B)80.1, interval 74.7 to 85.3, tier 2
  7. Kev-4B71.4, interval 65.2 to 77.4, tier 3
  8. Kev-0.6B60.6, interval 53.8 to 67.1, tier 4
Tested against Jev and 13 open models

Speed

Median latency, p50 · H100 only · lower is better

  1. Kev-9B43 ms
  2. Shisa DE-1
    46 ms
  3. Eikos-27B82 ms
  4. AutoJev-27B99 ms
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Shisa DE-1Other modelsTrained on the test data*Whisker: 95% interval