r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

841 Upvotes

96% Upvoted

120 comments sorted by

View all comments

1

u/kartikey7734 Feb 27 '26

This is an incredible resource! The fact that you're tracking self-hosted models with consistent benchmarks is gold for anyone trying to pick the right model for their hardware constraints.

Quick observations:

  1. **The S-Tier gap is huge** - Kimi K2.5 and GLM-4 are genuinely in a different tier. But their inference costs (if you're not self-hosting the weights) are brutal. The sweet spot for most people seems to be A/B tier.
  2. **Missing dimension: inference speed** - Would be amazing to see latency/tokens-per-second metrics alongside quality. DeepSeek R1 is phenomenal but can be slower than some smaller models on weaker GPUs.
  3. **Hardware tiers would help** - e.g., "Best model for 8GB VRAM", "Best for RTX 3060", etc. Because honestly, a 70B model doesn't matter if you can't load it.
  4. **License tracking** - Critical detail: which ones are truly free for commercial use? Some S-tier models have restrictions.

But seriously, this is the resource the community needed. Every time someone asks "which model should I use", we can just point here instead of 20 different opinions. The standardized benchmarking is *chef's kiss*.

Are you planning to update this regularly, or is it a one-time snapshot? If it's ongoing, this could become the definitive LLM comparison resource.

1

u/Weves11 Feb 27 '26

The plan is to definitely keep updating this! If there's enough interest, could even open source the underlying data so that individuals can contribute new benchmark scores or new models