LangWatch
Sign Up

InceptionProprietary model

Mercury Decide benchmarks

All models
  • General rank
    Too few tasks3 non-coding tasks ranked5 needed for a rank
  • Speed#6 / 8
    816 msMedian latency, p50Hosted API only
  • Cost#6 / 10
    $0.52Per 1,000 decisionsHosted price
  • Search#1 / 31 tied
    66.3%Best task · Accuracyvs Jev-class models

Comparison summary

Strongest
Judging and evals
Leads 2 of 3 (67%)

A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text. Strong at judging.

Specifications

Lab
Inception
Weights
Proprietary
Parameters
Not disclosed
Licence
Proprietary

Judging and evals

Set

Share of tasks led · #1 of 31

  1. Mercury Decide
    Leads 2 of 3
  2. OpenAI Decisions APILeads 4 of 6
  3. ClefLeads 1 of 3
  4. deck-31BLeads 1 of 3
  5. Kev-27BLeads 1 of 3
  6. pplx-decider v1.1 27BLeads 1 of 3
  7. Quyet-1.0-LargeLeads 1 of 3
  8. Rene-1 31B FP8Leads 1 of 3
See the 3 tasks

LLM-as-a-judge

Tier 2 of 2#2 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API71.7, interval 66.1 to 77.2, ahead alone
  2. Clef Flash63.0, interval 56.2 to 69.8, tier 2
  3. Mercury Decide
    61.7, interval 55.6 to 67.9, tier 2
  4. Decision 2.0 Eos 0.8B61.3, interval 54.6 to 68.0, tier 2
  5. Decision 2.0 Nox 4B61.3, interval 55.2 to 67.7, tier 2
  6. Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
  7. Jev61.3, interval 54.8 to 67.9, tier 2
  8. pplx-decider v1.1 27B61.3, interval 55.8 to 67.1, tier 2
Tested against Jev and 29 Jev-class models

Scenario judge

Tied for the lead#1 / 31

Accuracy, % · Higher is better

  1. Mercury Decide
    83.3, interval 75.5 to 90.3, tied for the lead
  2. OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
  3. deck-31B81.9, interval 74.8 to 88.1, tied for the lead
  4. Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
  5. pplx-decider v1.1 27B77.8, interval 69.8 to 84.9, tied for the lead
  6. Clef75.7, interval 67.4 to 83.1, tied for the lead
  7. Rene-1 31B FP875.0, interval 66.9 to 82.9, tied for the lead
  8. Kev-27B74.3, interval 66.4 to 82.0, tied for the lead
Tested against Jev and 29 Jev-class models

Search

Tied for the lead#1 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
  2. Mercury Decide
    66.3, interval 61.5 to 71.0, tied for the lead
  3. GLiDE63.9, interval 59.5 to 68.3, tier 2
  4. Jev63.3, interval 58.2 to 68.1, tier 2
  5. Kev-27B61.1, interval 56.4 to 65.7, tier 2
  6. Clef60.2, interval 55.7 to 64.9, tier 2
  7. deck-31B58.9, interval 54.3 to 63.5, tier 2
  8. Decision 2.0 Vega 27B58.3, interval 53.1 to 63.7, tier 2
Tested against Jev and 29 Jev-class models

Speed

Median latency, p50 · Hosted API only · lower is better

  1. OpenAI Decisions API227 ms
  2. Clef Omni330 ms
  3. Microsoft Decision 1371 ms
  4. Jev414 ms
  5. Clef756 ms
  6. Mercury Decide
    816 ms
  7. GLiDE860 ms
  8. Clef Flash1,007 ms

Cost

Per 1,000 decisions, hosted price · lower is better

  1. pplx-decider v1.1 27B$0.081
  2. AutoJev-27B$0.16
  3. Microsoft Decision 1$0.26
  4. Clef Flash$0.31
  5. Jev$0.43
  6. Mercury Decide
    $0.52
  7. OpenAI Decisions API$1.23
  8. GLiDE$1.30
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Mercury DecideOther modelsWhisker: 95% interval