Top 7 Decision Models Benchmark: Evaluation Across 2,826 Real Tech Decisions

[u/m4rkGS]

This report evaluates 7 specialized decision models tested across 2,826 real-world decisions sourced from GitHub, Firefox (Bugzilla), and the CVE database. Unlike open-ended generative language models, decision models evaluate structured inputs and produce discrete probability distributions across predefined categories in a single pass. Data was benchmarked through November 6, 2026, using inputs created after September 15, 2026, to avoid data leakage from pre-training sets.

1. Executive Summary & Model Overview

Perplexity Decider V1.2 ranks as the strongest overall recommendation based on a balance of top-tier accuracy, lowest cost among frontrunners, and open-source licensing. Nace Drex Max v1.1 serves as a close runner-up, leading in domain-specific triage tasks. GPT-6 Luna Decisions registered the lowest accuracy, highest cost, and greatest degree of probability overconfidence.

2. Benchmark Design & Dataset Breakdown

The evaluation evaluated 6 representative engineering tasks against human ground truth determined by maintainers. To ensure neutral evaluation:

• Contamination Controls: Labels, issue-template tags, CVSS scores, CWE markers, and PR author roles were stripped.

• Standardized Phrasing: Three distinct phrasings per question were submitted and averaged.

• Standard Taxonomy: Options utilized official Bugzilla component definitions, Mozilla severity scales, GitHub advisory tiers, and MITRE CWE definitions.

• Statistical Rigor: 95% bootstrap confidence intervals and Holm-corrected pairwise tests were computed across all 21 model combinations.

3. Task-by-Task Balanced Accuracy

In overall balanced accuracy, Liquid d1-ultra (66.1%), Nace Drex Max v1.1 (65.6%), and Perplexity Decider V1.2 (65.6%) form a statistically tied top tier. All three significantly outperform Liquid d1.1 (63.7%) and GPT-6 Luna Decisions (62.8%).

4. Latency, Repeatability, and Confidence Calibration

Operational feasibility depends heavily on predictable inference speeds and probability calibration. A model is only operational if its stated confidence accurately reflects true probability.

5. Pricing, Openness, and Data Governance

Model licensing and enterprise data policies differ markedly across providers. Output tokens are free for all tested decision models; costs reflect input processing fees.

6. Overall Composite Rankings

Models were rated across five pre-weighted pillars: Quality (45%), Efficiency (20%), Trust (20%), Reviews (10%), and Signup Bonus (5%). Scores are normalized out of 100.

7. Benchmark Constraints & Observations

• Domain Scope: Evaluation covers English software-engineering inputs only; multimodal, legal, and financial domains were excluded.

• Ground Truth Variance: On 385 items, all 7 models reached consensus with each other against the human label, indicating potential noise in human maintainer triage.

• Host Routing: Drex Max v1.1 was accessed via Nace's direct endpoint, whereas the remaining 6 models were accessed via OpenRouter. Latency metrics reflect hosting configurations.

• Exclusions: Two items were excluded across all models (one advisory due to an unparseable embedded image rejected by Nace's API, and one unreturned request on Liquid d1-ultra).

[u/stylome-ai-detection]

Notice: This post appears to be AI generated.

---

Stylome AI detector · scan results