GoogleProprietary model

Gemini Decision 1 benchmarks

All models
  • General rank
    Too few tasks3 non-coding tasks ranked5 needed for a rank
  • Speed#3 / 9
    370 msMedian latency, p50Hosted API only
  • Cost#9 / 11
    $1.29Per 1,000 decisionsHosted price
  • Scenario judge#1 / 32 tied
    80.6%Best task · Accuracyvs Jev-class models

Comparison summary

Strongest
Judging and evals
Leads 1 of 3 (33%)

A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text. Strongest at judging.

Strengths

  • Strong at judging simulated conversations from real use: as accurate as the best decision models, and it passes about nine in ten of the good ones

Weaknesses

  • Weak at searching real conversations: finds about one in three of the problems, fewer than most top models

Specifications

Lab
Google
Weights
Proprietary
Parameters
Not disclosed
Licence
Proprietary

Judging and evals

Set

Share of tasks led · #7 of 32

  1. Mercury DecideLeads 2 of 3
  2. OpenAI Decisions APILeads 4 of 6
  3. ClefLeads 1 of 3
  4. deck-31BLeads 1 of 3
  5. pplx-decider v1.1 27BLeads 1 of 3
  6. GLiDELeads 1 of 3
  7. Gemini Decision 1
    Leads 1 of 3
  8. Quyet-1.0-LargeLeads 1 of 3
See the 3 tasks

LLM-as-a-judge

Tier 2 of 3#3 / 32

Accuracy, % · Higher is better

  1. OpenAI Decisions API71.7, interval 66.1 to 77.2, tied for the lead
  2. Microsoft Decision 165.6, interval 59.0 to 72.5, tied for the lead
  3. Mercury Decide62.6, interval 56.6 to 69.1, tier 2
  4. Jev62.0, interval 56.1 to 68.3, tier 2
  5. Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
  6. pplx-decider v1.1 27B61.3, interval 56.0 to 67.0, tier 2
  7. Quyet-1.0-Large61.3, interval 55.8 to 67.3, tier 2
  8. Gemini Decision 1
    57.8, interval 51.3 to 65.1, tier 2

Scenario judge

Tied for the lead#1 / 32

Accuracy, % · Higher is better

  1. Mercury Decide83.3, interval 75.5 to 90.3, tied for the lead
  2. OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
  3. deck-31B81.9, interval 74.8 to 88.1, tied for the lead
  4. Gemini Decision 1
    80.6, interval 73.2 to 87.1, tied for the lead
  5. Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
  6. pplx-decider v1.1 27B76.4, interval 68.5 to 83.7, tied for the lead
  7. Clef75.7, interval 67.4 to 83.1, tied for the lead
  8. GLiDE75.0, interval 66.7 to 82.9, tier 2

Search

Tier 3 of 6#11 / 32

Accuracy, % · Higher is better

  1. OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
  2. Jev68.2, interval 63.0 to 73.1, tied for the lead
  3. Mercury Decide66.3, interval 61.5 to 71.0, tied for the lead
  4. GLiDE65.2, interval 60.9 to 69.4, tied for the lead
  5. Kev-27B61.1, interval 56.4 to 65.7, tier 2
  6. Clef60.2, interval 55.7 to 64.9, tier 2
  7. deck-31B58.9, interval 54.3 to 63.5, tier 2
  8. Gemini Decision 1
    53.5, interval 47.5 to 59.3, tier 3

Speed

Median latency, p50 · Hosted API only · lower is better

  1. OpenAI Decisions API225 ms
  2. Jev339 ms
  3. Gemini Decision 1
    370 ms
  4. Microsoft Decision 1390 ms
  5. Clef Omni455 ms
  6. Mercury Decide954 ms
  7. GLiDE961 ms
  8. Clef1,124 ms

Cost

Per 1,000 decisions, hosted price · lower is better

How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Gemini Decision 1Other modelsWhisker: 95% interval