LangWatch
Sign Up

GoogleProprietary model

Gemini 3.8 Flash benchmarks

All models
  • General rank
    #8of 27Leads 5 of 13 (38%)non-coding tasks
  • Speed#6 / 7
    1,936 msMedian latency, p50Hosted API only
  • Cost#3 / 8
    $0.57Per 1,000 decisionsHosted price
  • Moderation#1 / 8 tied
    81.4%Best task · Balanced accuracyvs frontier LLMs

Comparison summary

Strongest
Grounding
Leads 2 of 2 (100%)
Weakest
Safety and guardrails
Leads 1 of 4 (25%)

Strong at grounding, weaker at safety. It leads 5 of 13 (38%) non-coding tasks, #8 of 27 models by that count.

Specifications

Lab
Google
Weights
Proprietary
Parameters
Not disclosed
Licence
Proprietary

Overall standing

Share of non-coding tasks led · #8 of 27

  1. Claude Opus 5.585%
  2. Jev68%
  3. AutoJev-27B67%
  4. Eikos-27B60%
  5. GPT-6.1 Sol54%
  6. Claude Sonnet 5.546%
  7. Shisa DE-142%
  8. Gemini 3.8 Flash
    38%

Safety and guardrails

Share of tasks led · #5 of 8

  1. Claude Opus 5.5Leads 3 of 4
  2. JevLeads 8 of 12
  3. GPT-6.1 SolLeads 2 of 4
  4. Claude Sonnet 5.5Leads 2 of 4
  5. Gemini 3.8 Flash
    Leads 1 of 4
  6. Claude Haiku 4.5Leads 1 of 4
  7. GPT-5.6 TerraLeads 1 of 4
  8. GPT-6 LunaLeads 1 of 4
See the 4 tasks

Prompt injection

Tier 3 of 4#5 / 8

Balanced accuracy, % · Higher is better

  1. Jev95.2, interval 93.9 to 96.5, ahead alone
  2. Claude Haiku 4.592.7, interval 91.1 to 94.3, tier 2
  3. GPT-6.1 Sol91.3, interval 89.6 to 92.9, tier 2
  4. Claude Opus 5.590.9, interval 89.2 to 92.5, tier 2
  5. Gemini 3.8 Flash
    90.4, interval 88.6 to 92.1, tier 3
  6. Claude Sonnet 5.588.3, interval 86.4 to 90.2, tier 4
  7. GPT-5.6 Terra88.2, interval 86.3 to 90.0, tier 4
  8. GPT-6 Luna88.0, interval 86.1 to 89.8, tier 4
Tested against Jev and 6 frontier LLMs

Moderation

Tied for the lead#1 / 8

Balanced accuracy, % · Higher is better

  1. Claude Sonnet 5.581.6, interval 78.5 to 84.4, tied for the lead
  2. GPT-6.1 Sol81.4, interval 78.5 to 84.2, tied for the lead
  3. Gemini 3.8 Flash
    81.4, interval 78.4 to 84.1, tied for the lead
  4. Claude Opus 5.581.1, interval 78.0 to 84.0, tied for the lead
  5. Claude Haiku 4.581.1, interval 78.0 to 84.0, tied for the lead
  6. GPT-6 Luna80.8, interval 77.6 to 83.7, tied for the lead
  7. GPT-5.6 Terra79.9, interval 77.0 to 82.9, tied for the lead
  8. Jev78.0, interval 74.9 to 81.0, tier 2
Tested against Jev and 6 frontier LLMs

Off-topic

Tier 2 of 4#4 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.596.0, interval 94.8 to 97.1, tied for the lead
  2. Claude Sonnet 5.595.4, interval 94.1 to 96.6, tied for the lead
  3. GPT-6.1 Sol95.3, interval 93.9 to 96.6, tied for the lead
  4. Gemini 3.8 Flash
    95.0, interval 93.6 to 96.3, tier 2
  5. Jev93.4, interval 91.8 to 94.9, tier 3
  6. GPT-5.6 Terra92.3, interval 90.6 to 93.9, tier 3
  7. Claude Haiku 4.591.5, interval 89.7 to 93.2, tier 4
  8. GPT-6 Luna90.6, interval 88.7 to 92.4, tier 4
Tested against Jev and 6 frontier LLMs

PII

Tier 2 of 5#2 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.595.4, interval 94.1 to 96.6, ahead alone
  2. Claude Sonnet 5.594.0, interval 92.6 to 95.5, tier 2
  3. Gemini 3.8 Flash
    93.1, interval 91.5 to 94.6, tier 2
  4. Jev92.9, interval 91.3 to 94.4, tier 2
  5. GPT-6.1 Sol91.3, interval 89.7 to 93.0, tier 3
  6. GPT-6 Luna87.2, interval 85.3 to 89.1, tier 4
  7. GPT-5.6 Terra86.4, interval 84.5 to 88.4, tier 4
  8. Claude Haiku 4.583.1, interval 80.9 to 85.3, tier 5
Tested against Jev and 6 frontier LLMs

Routing and classification

Share of tasks led · #4 of 8

  1. Claude Opus 5.5Leads 3 of 4
  2. JevLeads 9 of 14
  3. GPT-5.6 TerraLeads 2 of 4
  4. Gemini 3.8 Flash
    Leads 1 of 4
  5. Claude Sonnet 5.5Leads 1 of 4
  6. GPT-6.1 SolLeads 1 of 4
  7. GPT-6 LunaLeads 1 of 4
  8. Claude Haiku 4.5Leads 0 of 4
See the 4 tasks

Routing, 20 intents

Tier 2 of 3#2 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.595.0, interval 93.6 to 96.3, ahead alone
  2. GPT-6.1 Sol92.8, interval 91.2 to 94.3, tier 2
  3. Gemini 3.8 Flash
    92.7, interval 91.1 to 94.2, tier 2
  4. GPT-5.6 Terra92.4, interval 90.7 to 94.0, tier 2
  5. Claude Sonnet 5.591.7, interval 89.9 to 93.4, tier 2
  6. GPT-6 Luna90.3, interval 88.4 to 92.1, tier 3
  7. Claude Haiku 4.589.3, interval 87.3 to 91.2, tier 3
  8. Jev89.1, interval 87.0 to 91.0, tier 3
Tested against Jev and 6 frontier LLMs

Routing, 77 intents

Tier 2 of 4#2 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.588.0, interval 85.2 to 90.8, ahead alone
  2. Gemini 3.8 Flash
    84.8, interval 81.6 to 88.0, tier 2
  3. GPT-6.1 Sol84.4, interval 81.2 to 87.6, tier 2
  4. Claude Sonnet 5.584.2, interval 81.0 to 87.2, tier 2
  5. GPT-5.6 Terra83.4, interval 80.0 to 86.6, tier 2
  6. GPT-6 Luna82.2, interval 78.8 to 85.6, tier 3
  7. Claude Haiku 4.580.0, interval 76.6 to 83.4, tier 3
  8. Jev79.6, interval 76.0 to 83.0, tier 4
Tested against Jev and 6 frontier LLMs

Tool routing

Tier 2 of 4#3 / 8

Accuracy, % · Higher is better

  1. GPT-6 Luna85.5, interval 83.3 to 87.6, tied for the lead
  2. GPT-5.6 Terra84.2, interval 82.0 to 86.3, tied for the lead
  3. Gemini 3.8 Flash
    83.6, interval 81.3 to 85.7, tier 2
  4. Claude Haiku 4.582.0, interval 79.6 to 84.2, tier 2
  5. GPT-6.1 Sol79.9, interval 77.4 to 82.2, tier 3
  6. Claude Sonnet 5.579.4, interval 76.9 to 81.7, tier 3
  7. Claude Opus 5.578.7, interval 76.1 to 81.1, tier 4
  8. Jev78.3, interval 75.7 to 80.7, tier 4
Tested against Jev and 6 frontier LLMs

Complaint routing

Tied for the lead#1 / 8

Accuracy, % · Higher is better

  1. Gemini 3.8 Flash
    79.6, interval 77.1 to 82.0, tied for the lead
  2. GPT-6.1 Sol79.0, interval 76.5 to 81.4, tied for the lead
  3. Claude Opus 5.578.9, interval 76.4 to 81.4, tied for the lead
  4. Jev78.7, interval 76.3 to 81.2, tied for the lead
  5. GPT-5.6 Terra78.4, interval 75.9 to 81.0, tied for the lead
  6. Claude Sonnet 5.578.3, interval 75.7 to 80.8, tied for the lead
  7. GPT-6 Luna77.9, interval 75.4 to 80.4, tier 2
  8. Claude Haiku 4.577.1, interval 74.6 to 79.7, tier 2
Tested against Jev and 6 frontier LLMs

Grounding

Share of tasks led · #3 of 8

  1. Claude Opus 5.5Leads 2 of 2
  2. Claude Sonnet 5.5Leads 2 of 2
  3. Gemini 3.8 Flash
    Leads 2 of 2
  4. GPT-6.1 SolLeads 2 of 2
  5. JevLeads 4 of 6
  6. GPT-5.6 TerraLeads 0 of 2
  7. GPT-6 LunaLeads 0 of 2
  8. Claude Haiku 4.5Leads 0 of 2
See the 2 tasks

RAG faithfulness

Tied for the lead#1 / 8

Balanced accuracy, % · Higher is better

  1. Claude Opus 5.585.0, interval 82.3 to 87.5, tied for the lead
  2. GPT-6.1 Sol84.2, interval 81.5 to 86.9, tied for the lead
  3. Claude Sonnet 5.583.6, interval 80.8 to 86.4, tied for the lead
  4. Gemini 3.8 Flash
    83.3, interval 80.4 to 86.2, tied for the lead
  5. GPT-5.6 Terra80.6, interval 77.7 to 83.6, tier 2
  6. Jev80.3, interval 77.5 to 83.1, tier 2
  7. GPT-6 Luna79.7, interval 76.7 to 82.7, tier 2
  8. Claude Haiku 4.577.4, interval 74.4 to 80.3, tier 3
Tested against Jev and 6 frontier LLMs

Search relevance

Tied for the lead#1 / 8

Accuracy, % · Higher is better

  1. Claude Opus 5.563.6, interval 60.7 to 66.7, tied for the lead
  2. GPT-6.1 Sol63.0, interval 60.0 to 66.1, tied for the lead
  3. Gemini 3.8 Flash
    62.5, interval 59.6 to 65.6, tied for the lead
  4. Claude Sonnet 5.561.1, interval 58.1 to 64.0, tied for the lead
  5. Jev57.7, interval 54.7 to 60.8, tier 2
  6. GPT-5.6 Terra57.6, interval 54.3 to 60.6, tier 2
  7. GPT-6 Luna54.7, interval 51.6 to 57.8, tier 2
  8. Claude Haiku 4.550.6, interval 47.4 to 53.7, tier 3
Tested against Jev and 6 frontier LLMs

Judging and evals

Set

Share of tasks led · #5 of 5

  1. Claude Opus 5.5Leads 3 of 3
  2. GPT-6.1 SolLeads 2 of 3
  3. JevLeads 3 of 6
  4. Claude Sonnet 5.5Leads 1 of 3
  5. Gemini 3.8 Flash
    Leads 1 of 3
See the 3 tasks

LLM-as-a-judge

Tier 2 of 3#3 / 5

Accuracy, % · Higher is better

  1. GPT-6.1 Sol92.6, interval 89.1 to 95.8, tied for the lead
  2. Claude Opus 5.589.1, interval 83.9 to 93.9, tied for the lead
  3. Claude Sonnet 5.583.5, interval 78.4 to 88.5, tier 2
  4. Gemini 3.8 Flash
    80.9, interval 76.2 to 85.5, tier 2
  5. Jev61.3, interval 54.8 to 67.9, tier 3
Tested against Jev and 3 frontier LLMs

Scenario judge

Tied for the lead#1 / 5

Accuracy, % · Higher is better

  1. Claude Opus 5.586.8, interval 80.2 to 92.3, tied for the lead
  2. Gemini 3.8 Flash
    83.3, interval 76.8 to 89.1, tied for the lead
  3. Claude Sonnet 5.581.3, interval 74.1 to 87.4, tier 2
  4. GPT-6.1 Sol81.3, interval 74.1 to 87.3, tier 2
  5. Jev66.7, interval 57.3 to 75.3, tier 3
Tested against Jev and 3 frontier LLMs

Search

Tier 2 of 3#4 / 5

Accuracy, % · Higher is better

  1. Claude Opus 5.591.5, interval 87.6 to 94.8, tied for the lead
  2. GPT-6.1 Sol89.6, interval 85.6 to 93.1, tied for the lead
  3. Claude Sonnet 5.589.1, interval 85.8 to 92.1, tied for the lead
  4. Gemini 3.8 Flash
    85.0, interval 81.3 to 88.4, tier 2
  5. Jev63.3, interval 58.2 to 68.1, tier 3
Tested against Jev and 3 frontier LLMs

Speed

Median latency, p50 · Hosted API only · lower is better

  1. Jev~65 ms
  2. GPT-6 Luna1,124 ms
  3. Claude Haiku 4.51,306 ms
  4. GPT-5.6 Terra1,565 ms
  5. Claude Sonnet 5.51,646 ms
  6. Gemini 3.8 Flash
    1,936 ms
  7. Claude Opus 5.52,194 ms

Cost

Per 1,000 decisions, hosted price · lower is better

  1. Jev$0.029
  2. GPT-6 Luna$0.049
  3. Gemini 3.8 Flash
    $0.57
  4. Claude Haiku 4.5$0.69
  5. GPT-6.1 Sol~$1.02
  6. GPT-5.6 Terra$1.03
  7. Claude Sonnet 5.5$1.52
  8. Claude Opus 5.5$3.44
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Gemini 3.8 FlashOther modelsWhisker: 95% interval