GoogleProprietary model
Gemini Decision 1 benchmarks
- General rankToo few tasks3 non-coding tasks ranked5 needed for a rank
- Speed#3 / 9370 msMedian latency, p50Hosted API only
- Cost#9 / 11$1.29Per 1,000 decisionsHosted price
- Scenario judge#1 / 32 tied80.6%Best task · Accuracyvs Jev-class models
Comparison summary
- Strongest
- Judging and evals
- Leads 1 of 3 (33%)
A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text. Strongest at judging.
Strengths
- Strong at judging simulated conversations from real use: as accurate as the best decision models, and it passes about nine in ten of the good ones
Weaknesses
- Weak at searching real conversations: finds about one in three of the problems, fewer than most top models
Specifications
- Lab
- Weights
- Proprietary
- Parameters
- Not disclosed
- Licence
- Proprietary
Judging and evals
Share of tasks led · #7 of 32
- Mercury DecideLeads 2 of 3
- OpenAI Decisions APILeads 4 of 6
- ClefLeads 1 of 3
- deck-31BLeads 1 of 3
- pplx-decider v1.1 27BLeads 1 of 3
- GLiDELeads 1 of 3
- Gemini Decision 1Leads 1 of 3
- Quyet-1.0-LargeLeads 1 of 3
See the 3 tasks
LLM-as-a-judge
Tier 2 of 3#3 / 32Accuracy, % · Higher is better
- OpenAI Decisions API71.7, interval 66.1 to 77.2, tied for the lead
- Microsoft Decision 165.6, interval 59.0 to 72.5, tied for the lead
- Mercury Decide62.6, interval 56.6 to 69.1, tier 2
- Jev62.0, interval 56.1 to 68.3, tier 2
- Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
- pplx-decider v1.1 27B61.3, interval 56.0 to 67.0, tier 2
- Quyet-1.0-Large61.3, interval 55.8 to 67.3, tier 2
- Gemini Decision 157.8, interval 51.3 to 65.1, tier 2
Scenario judge
Tied for the lead#1 / 32Accuracy, % · Higher is better
- Mercury Decide83.3, interval 75.5 to 90.3, tied for the lead
- OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
- deck-31B81.9, interval 74.8 to 88.1, tied for the lead
- Gemini Decision 180.6, interval 73.2 to 87.1, tied for the lead
- Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
- pplx-decider v1.1 27B76.4, interval 68.5 to 83.7, tied for the lead
- Clef75.7, interval 67.4 to 83.1, tied for the lead
- GLiDE75.0, interval 66.7 to 82.9, tier 2
Search
Tier 3 of 6#11 / 32Accuracy, % · Higher is better
- OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
- Jev68.2, interval 63.0 to 73.1, tied for the lead
- Mercury Decide66.3, interval 61.5 to 71.0, tied for the lead
- GLiDE65.2, interval 60.9 to 69.4, tied for the lead
- Kev-27B61.1, interval 56.4 to 65.7, tier 2
- Clef60.2, interval 55.7 to 64.9, tier 2
- deck-31B58.9, interval 54.3 to 63.5, tier 2
- Gemini Decision 153.5, interval 47.5 to 59.3, tier 3
Speed
Median latency, p50 · Hosted API only · lower is better
- OpenAI Decisions API225 ms
- Jev339 ms
- Gemini Decision 1370 ms
- Microsoft Decision 1390 ms
- Clef Omni455 ms
- Mercury Decide954 ms
- GLiDE961 ms
- Clef1,124 ms
Cost
Per 1,000 decisions, hosted price · lower is better
- pplx-decider v1.1 27B$0.064
- AutoJev-27B$0.16
- Clef Flash$0.21
- Microsoft Decision 1$0.25
- Jev$0.43
- Mercury Decide$0.52
- OpenAI Decisions API$1.23
- Gemini Decision 1$1.29
How we count
A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.
Gemini Decision 1Other modelsWhisker: 95% interval

