LangWatchOpen weights model29.4B parameters

Vega 27B Fine-Tuned (LangWatch) benchmarks

All models
  • Cost
    Self-hostedHardware not recordedNo hosted price

Comparison summary

A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text.

Specifications

Lab
LangWatch
Weights
Open weights
Parameters
29.4B
Licence
Not recorded

Judging and evals

Set

Tested on 3 tasks here, none ranked.

See the 3 tasks

LLM-as-a-judge

Not ranked

Accuracy, % · Higher is better

  1. OpenAI Decisions API71.7, interval 66.1 to 77.2, tied for the lead
  2. Vega 27B Fine-Tuned (LangWatch)
    67.0, interval 59.9 to 74.5, not ranked
  3. Microsoft Decision 165.6, interval 59.0 to 72.5, tied for the lead
  4. Mercury Decide62.6, interval 56.6 to 69.1, tier 2
  5. Jev62.0, interval 56.1 to 68.3, tier 2
  6. Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
  7. pplx-decider v1.1 27B61.3, interval 56.0 to 67.0, tier 2
  8. Quyet-1.0-Large61.3, interval 55.8 to 67.3, tier 2

Scenario judge

Not ranked

Accuracy, % · Higher is better

  1. Mercury Decide83.3, interval 75.5 to 90.3, tied for the lead
  2. OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
  3. deck-31B81.9, interval 74.8 to 88.1, tied for the lead
  4. Gemini Decision 180.6, interval 73.2 to 87.1, tied for the lead
  5. Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
  6. pplx-decider v1.1 27B76.4, interval 68.5 to 83.7, tied for the lead
  7. Clef75.7, interval 67.4 to 83.1, tied for the lead
  8. Vega 27B Fine-Tuned (LangWatch)
    74.3, interval 65.4 to 82.0, not ranked

Search

Not ranked

Accuracy, % · Higher is better

  1. Vega 27B Fine-Tuned (LangWatch)
    75.6, interval 70.8 to 80.3, not ranked
  2. OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
  3. Jev68.2, interval 63.0 to 73.1, tied for the lead
  4. Mercury Decide66.3, interval 61.5 to 71.0, tied for the lead
  5. GLiDE65.2, interval 60.9 to 69.4, tied for the lead
  6. Kev-27B61.1, interval 56.4 to 65.7, tier 2
  7. Clef60.2, interval 55.7 to 64.9, tier 2
  8. deck-31B58.9, interval 54.3 to 63.5, tier 2
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Vega 27B Fine-Tuned (LangWatch)Other modelsWhisker: 95% interval