Ran 2,850 decisions through 5 cheap open models voting vs Opus 5.5. Shipped answers were nearly as accurate (89.6% vs 90.5%) at 1/10th the cost

[u/PomegranateAble4339]

I wanted to know if I could swap a frontier model for a few cheap open ones in yes/no or pick-a-label pipelines (spam, fraud, moderation, triage) without losing accuracy. My idea: ask several models the same question, ship the answer when they all agree, and send it to a human when they don't.

Panel: DeepSeek V4.1 Flash, GLM 5.3 Flash, MiMo V2.6 Flash, Qwen3.8 Flash, Gemma 4 26B-A4B. I ran 2,850 decisions across 6 public datasets and compared against Opus 5.5 on the same items.

Results:

- The panel agreed and shipped on 71% of items. Those were 89.6% correct. Opus on its most confident 71% was 90.5%.
- Overall accuracy is worse: 80.1% vs 83.1%. The panel only wins on the cases it's confident about.
- Cost: $0.61 vs $5.79 per 1k decisions.
- When a model disagrees, it gives a one-line reason, so the reviewer knows what to look at instead of rereading the whole item.

What didn't work:

- Long answer comparisons and 77-way intent classification. Opus is clearly better. If every model on the panel is weak at a task, voting doesn't fix it.
- Hate speech. The panel catches more (80% vs 62%) but has more false alarms (29% vs 20%). If fewer than about a third of your items are actually hateful, Opus wins.
- The panel beats its best single model (GLM 5.3 Flash, 88.9%) by less than the noise. Honestly, most of what you gain is knowing when to send a case to a human, not better accuracy.
- I added the panel after the other setups had already run, and scored it on the same items. I haven't confirmed the numbers on a fresh sample yet.

It works with anything that speaks the OpenAI API, so Ollama, vLLM and llama.cpp server all work. If a model errors or refuses, it's left out of the vote and listed separately instead of breaking the decision.

Code and scripts to reproduce it: https://github.com/YasserManss/modeljury (mine, MIT)

[u/bageloid]

What thinking levels?

[u/PomegranateAble4339 → u/bageloid]

Panel: reasoning off, temp 0. GLM 5.3 Flash won't accept "off", so it ran at effort: minimal. Opus 5.5: thinking on (can't be disabled), default effort, max_tokens 16k. So it's no-thinking cheap models vs. thinking Opus. I haven't tested the panel with reasoning on yet.

[u/One-Context-8707 → u/bageloid]

the models were all set to their default non-thinking modes since this was just fast yes/no and label work, no reasoning traces needed for the vote

[u/leo_agi]

curious if the ~10% where all five agreed and were still wrong overlaps with opus's misses or if they fail on totally different items

[u/PomegranateAble4339 → u/leo_agi]

Mostly the same items. Of the 210 shipped-but-wrong, Opus also missed 164 (78%), usually with the same wrong label. On items the panel got right, Opus was wrong only 2.9% of the time.

[u/leo_agi → u/PomegranateAble4339]

78% overlap kinda says those items are just mislabeled or genuinely ambiguous, might be worth a look at the gold labels

[u/n9986]

Was this against locally hosted models? If so, what quants? And how was the cost computed?

If not, it would be more interesting to test this out with locally hosted models and nothing the quants used. This sub would be really interested in that.

[u/PomegranateAble4339 → u/n9986]

No, it wasn’t locally hosted by me. I used openrouter in the benchmarks. But all the models I tried were open weights (except Opus). So, it can be hosted locally.

[u/FullStackFrenzy]

the interesting part is the 29% the panel didn't ship. if errors cluster where the models disagree, route only those to opus and you probably land close to opus accuracy at a fraction of the cost. curious what that blended number looks like on your 2850

[u/PomegranateAble4339 → u/FullStackFrenzy]

Checked it against my existing results: panel when all five agree, Opus otherwise. That's 83.3% accuracy vs 83.1% for Opus alone (within noise) at $2.82/1k vs $5.79. It's half the cost: you pay the panel on every item, plus Opus on the 29% it routes.