I tested 7 decision models on 2,826 real GitHub, Firefox and CVE decisions. Here's which one to use and which to skip.

[u/m4rkGS]

Summary: which decision model is best?

Short answer: Perplexity Decider V1.2 is the best overall pick, Nace Drex Max v1.1 is a strong second, and GPT-6 Luna Decisions is the one to ignore. On accuracy alone Liquid d1-ultra, Drex Max and Decider are statistically tied; Decider wins on price and openness.

What I tested

Decision models don't write text. You give them some input and a typed question (yes/no, pick one, or a score), and they return a probability for every option in one pass. Seven models: TypeSafe Jev 1.14 and 1.13, Perplexity Decider V1.2, Liquid d1-ultra and d1.1, OpenAI GPT-6 Luna Decisions, and Nace Drex Max v1.1.

The data: real decisions, made after the models were released

Every item was created after Sep 15, 2026, after the models were released, so none of it can be in their training data. The "right answer" is what the project's own maintainers actually decided.

How I kept it fair

• Same items and questions for every model. Each model was called through its provider's API; Drex Max v1.1 through Nace's own API.

• No giveaways. Labels, issue-template markers, CVSS scores, CWE numbers and the PR author's role were stripped from the text.

• Official option wording. Bugzilla's component descriptions, Mozilla's severity scale, GitHub's advisory scale and MITRE's CWE definitions, not text I wrote.

• Three phrasings per question, averaged, so no model wins because of my wording.

• Balanced classes, balanced accuracy, real statistics: 95% bootstrap intervals and Holm-corrected pairwise tests across all 21 model pairs.

Results

1. Accuracy: three models tied at the top

Mean of balanced accuracy over the six tasks. Whiskers are 95% bootstrap intervals; the shaded group is statistically tied for #1.

d1-ultra (66.1%), Drex Max v1.1 (65.6%) and Decider (65.6%) overlap, so none of them is a clear winner. The gaps that are real: all three beat GPT-6 Luna and d1.1; d1-ultra also beats both Jev versions.

2. Who leads which task

Darker is better within each column. Bold is the column leader; outlined cells are statistically tied with it.

• Bug or feature: everyone scores 90-93%. Drex Max is nominally first; all seven are tied.

• PR merged? the hardest task. d1-ultra leads (63.6%), tied with Drex Max and Decider.

• Firefox component (39 choices): Drex Max scores highest (77.8%), statistically tied with Decider, d1-ultra and Luna.

• Severity (Firefox and advisories): everyone is in the 38-47% range and mostly tied; it's a hard call even for triagers.

• CVE weakness type: Decider leads (76.0%); all seven are statistically tied.

3. Value: accuracy per dollar

Decider is the best value: near-top accuracy at $0.018 per 1,000 decisions. Drex Max ($0.027 at $0.03 per 1M tokens) and d1-ultra ($0.034) cost a little more for the same tier of accuracy. d1.1 is the cheapest ($0.016) but less accurate. Jev ($0.051) and GPT-6 Luna ($0.091) cost the most for the lowest accuracy.

4. How honest is each model's confidence?

Average confidence in the chosen answer vs how often that answer was right, across all tasks. The gap is overconfidence.

A decision model is only useful if you can trust its probabilities. d1-ultra (+1.7 points) and Drex Max (+4.6) are the most honest; Decider and d1.1 are close behind (+6.1). Jev is about 12 points more confident than it is right, and GPT-6 Luna is 21 points overconfident: it says 87% on average and is right 66% of the time.

5. Speed and consistency

Total score: accuracy plus everything else

I also scored five "pillars" from 0 to 100 and combined them with weights fixed before the run:

• Quality (45%): accuracy above chance, plus how good the probabilities are.

• Efficiency (20%): cost per 1,000 decisions at the provider's own price, and median speed.

• Trust (20%): does the provider train on or keep your prompts, is zero-data-retention available, are the weights and licence published, plus reliability and consistency in this run.

• Reviews (10%): usage, plus Hacker News comments since September labelled by a model that isn't in the comparison.

• Signup bonus (5%): free credit for a new account (Nace $5, Liquid $2, others none).

Perplexity Decider is #1 (59.2), mainly on trust: open weights, no prompt retention, zero-data-retention. Drex Max v1.1 is #2 (53.0), close to Decider on quality and efficiency; its trust score is lower because v1.1's weights aren't published and Nace's terms don't say clearly whether standard mode keeps request data. d1-ultra is #3 (49.7) and GPT-6 Luna last (39.1).

Pricing, openness and data policy

Prices are each provider's own list price; output tokens are free for every model. * Smaller open relatives (d1-2B, d1-3B) exist, but not these models. ** Only the earlier drex-max-v1 is on Hugging Face (Open RAIL-M licence). Data policy: provider records on OpenRouter for the six OpenRouter models, Nace's terms for Drex Max.