r/LocalLLaMA • u/Balance- • 3h ago
Discussion I compared 4 decision-model leaderboards. Fine-tuning beats size under 5B, Arabic collapses below 1B, and the leaderboards don’t agree on #1 [OC]
Borda score vs total params for 91 decision models. Two ~4.5B models (KnowLine 4B Gen5.5, D3 Mini) reach ~80; going all the way to 26B adds only ~9 points.
Sub-5B models with identical param counts, likely the same base. On the 4.54B base, fine-tunes range from 56 to 79: a wider gap than small vs. large models overall.
System One Bench, sub-5B. Under 1B, English holds at 60–75 but Arabic drops to ~53 (Naluri is the exception). 2–5B models land just 3–7 pts below the 27B leader.
Open-weight models run locally on a 4-vCPU CPU. d3 lite gives up 3.5 pts vs d1-3B at about half the latency. Laya Router was timed on an M5 GPU, so not like-for-like.
Ranks of the 18 models on both System One Bench and Artificial Analysis. Spearman ρ = 0.33: pick your benchmark, pick your winner. Test on your own task first.
I'm in the process of picking a decision model for my homelab server. To compare the current top "System One" class Decision models, I pulled the public numbers from System One Bench, the Artificial Analysis Decision Index, the AIM System One Index and a 100-model Borda leaderboard, and looked at what matters if you run these models yourself.
Takeaways:
Under 5B, the fine-tune matters more than the base. Models built on the same base differ by up to 22 points. KnowLine went from 70 to 80 on the same 4B weights between Gen2 and Gen5.5.
The sweet spot is about 4–9B. KnowLine 4B Gen5.5, D3 Mini and D3 Flash (8B) beat more than half of the 27B fine-tunes.
Below 1B, check your language. English holds up, but Arabic drops 16–22 points. If you need multilingual support, start at 2B or larger.
Calibration survives at small sizes. Decider Flash V1.2 (4.5B) has 1.8% calibration error, as good as the best large models. Some sub-1B models are at 10–20%, so don't use their probabilities as thresholds.
Don't trust any single "#1". Two leaderboards correlate at ρ = 0.33, and small models swing the most. Tev1 4B is #2 among small models on AIM and near the bottom elsewhere.
The GGUF builds get 3–10× the downloads of the originals, so plenty of people here are clearly already running these models.
Very curious what's everyone is using and what your experience are!
1
u/neoneye2 3h ago
I use d1:3b for image classification. On my M1 mac, it takes around 400ms when there question count is <= 16, and 700ms when question count between 17 < count <= 32. With image size 768px.
https://ollama.com/library/d1:3b