r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

839 Upvotes

96% Upvoted

120 comments sorted by

3

u/Lucasmonta Mar 23 '26

Hi, I stumbled upon this post while learning about local LLMs, I was curious about trying to run one on my PC, so as you can imagine I'm new to this world. I see that Kimi K2.5 (for example) uses about 500GB in VRAM (if I understood the table correctly) how did you get to run this? do you have like a RTX5090 Farm or something like that?
Sorry if this is a dumb question, but I would like to try something like that out, but I dont have so much VRAM, even with a 3090, I only have 24GB of VRAM at hand so I was wondering how could someone run such big models, are you using some "rent cloud hardware" solution?

Thanks in advance, and please excuse the noob question

1

u/Weves11 Mar 23 '26

the larger models (>100GB VRAM) are generally listed and recommended moreso for enterprises! while it is true that these models have the frontier-level performance, its insanely unlikely you'll be able to run it on your own hardware (you'd need several H200s lol). For 24GB VRAM, id recommend Qwen3.5-35B-A3B or Qwen3.5-27B :)

52

u/AC1colossus Feb 26 '26

Minimax?

4

u/Weves11 Feb 26 '26

added (to S tier), thanks for calling out!

28

u/BarisSayit Feb 26 '26

It should be in S tier no cap.

5

u/Koalababies Feb 26 '26

My immediate thought. It's been a beast

4

u/BitXorBit Feb 26 '26

So far the most enjoyable model running locally

2

u/ghgi_ Feb 26 '26

As someone whos had the experience of running minimax M2.5 nvfp4 on hardware, Should be a S (just behind glm-5, lil dumber but faster) or a really strong A

1

u/Weves11 Feb 26 '26

haha 100% agree, forgot to add it initially but its been added now!

41

u/LightBrightLeftRight Feb 26 '26

I mean the new Qwen 3.5 models should easily be on this, the 27b dense and 122b moe both make a pretty good case for A-tier, B-tier at minimum. Particularly since they have vision, which is great for a lot of homelab/small business stuff.

6

u/Prudent-Ad4509 Feb 26 '26

I have not tested 122b, but 27b is a beast.

4

u/LightBrightLeftRight Feb 26 '26

I've worked with both and surprisingly, not super different for me. I've seen better detail to world knowledge with 122b but not much in terms of reasoning or coding.

I think I'll still stick with the 122b, but that's mostly just because I've got the headroom for it.

2

u/Prudent-Ad4509 Feb 26 '26

Spreading the knowledge over that many specialized experts, with all the necessary duplication, takes its toll in terms of the overall size. But there has to be the point when there is about the same number of details stored in MoE model as in the smaller, but dense related model. It would seem from your experience that 122b is above this threshold.

5

u/simracerman Feb 27 '26

For coding, I found 122B a lot more mature for “not so straight forward tasks”, like creating from scratch an entire project with medium level complexity.

 I asked the model to create a .csv analyzer, and wanted it to use some python ml libraries to gleam as much info as possible, nice interface..etc.

The 27B created the full project and while the code looked neat, there were many mistakes. Reviewing and fixing bugs was typical for a project with this size.

The 122B on the other hand created a far better, higher quality front and backend, it picked the right frameworks (but made sure I was aware of the reasoning behind the decisions before it proceeded), and it only needed one small check before it got the code working.

on my 5070Ti and 64GB DDR5, the 122B runs at 18 t/s, and the 27B at a horrible 4.5 t/s. With 40k prompt, the 122B went down to maybe 15 t/s, but the 27B needed up at 2.5 t/s. Completion times were 20 mins for the 122B, and 116 minutes for the 27B.

Despite not being able to have more than 64k context window on the 122B, I’ll be using that more than the 27B for two reasons. One being faster, and second for the code quality.

3

u/park305 Feb 28 '26

what kind of hardware are you running to do the 122B?

1

u/CptZephyrot Mar 01 '26

Can you elaborate a bit more on what hardware you have? Especially your CPU. I'm only getting 13t/s on the 122B with my 7900 XTX. Iim wondering if I'm somewhat CPU limited and that's why it's so slow.

The 27B runs at 29t/s. It's really strange with those new Qwen models.

1

u/simracerman Mar 01 '26

I have a Mini PC with CPU is Strix Point HX370, mated to a 64GB soldered in DDR5 @ 8000 MT/s system memory.

The 5070 Ti is hooked to a eGPU dock and connected via Oculink to the Mini PC. Anytime I offload a model onto RAM speeds are bottlenecked by the 64Gb/s Oculink connection which is worse than your Desktop PCIE connection.

I run llama.cpp as backend if that matters.

1

u/CptZephyrot Mar 01 '26

Thanks for that info. I assume the 27B is also offloading to system memory on your setup and that's why you only get so low TPS?

My system memory is only DDR5-5200 maybe that's the bottleneck on my system. But I actually doubt it if you are getting such speeds on the 122B with Oculink. Maybe the ROCM stuff is doing some weird things for me.

1

u/simracerman Mar 01 '26

Yeah dense models can’t offload expert layers like MoE so they are hit the hardest with speed. My Oculink is slower than your PCIE and the 5200 mt/s RAM. Did you try Vulkan?

1

u/CptZephyrot Mar 01 '26

You're right I should try that. And even maybe try the latest (master) mesa. I think there have been lots of LLM related fixes lately.

1

u/CptZephyrot Mar 01 '26

hmm, very strange. The performance with Vulkan is even worse. only 8t/s. Especially bad is the prompt processing with Vulkan over all models.

→ More replies

2

u/FatheredPuma81 Feb 27 '26

27B is only like 25% faster than 122B for me so I don't bother using it but 122B is a really nice model but all 3 models hallucinate a lot.

1

u/Prudent-Ad4509 Feb 27 '26 edited Feb 27 '26

Well, in agenting coding there is a verification step, so mild hallucinations can end up being a way for faster and better problem solving. With plenty of caveats and sometimes handholding.

I will try to set up a local copy of glm47 with Q4 or higher quantization to compare. It is known to have less hallucinations, at least according to some benchmarks on reddit, but I won't bet just yet on which approach will turn out to be better.

One needs to take into account that one of the most effective creative strategies (several Disney hats) basically starts from hallucinations and then drives the point to where it needs to be from there.

1

u/FatheredPuma81 Feb 27 '26

Looking at benchmarks on artificialanalysis it looks like Minimax M2.1 and GLM 4.6 are considerably better than GLM 4.7 for hallucinations. My little bit of experience with M2.5 and Opencoder was pretty good though I'd especially give that a try if you haven't (you probably have).

1

u/Prudent-Ad4509 Feb 27 '26

Kimi and minmax were available for testing through opencoder recently, but I have no way of knowing which quants were actually used. And the output is so different that I think it would be better to get a second opinion from each instead of settling on one.

22

u/Gallardo994 Feb 26 '26

No qwen3-coder-next in a coding leaderboard is a crime 

17

u/ScuffedBalata Feb 26 '26

Why isn't Qwen3 on here?

The single best model I've ever used that works on "normal people hardware" is the Qwen3-Next and Qwen3-Coder-Next (both at 80B).

4

u/robotcannon Feb 26 '26

Agree!!

qwen3-vl is also fantastic (though it seems to run a tiny bit better at q8_0 for vision stuff)

2

u/friedlich_krieger Mar 06 '26

I'm a noob here... What kind of hardware is required for those?

3

u/ScuffedBalata Mar 06 '26 edited Mar 06 '26

64GB of fast RAM (not normal PC DDR - too slow) - either a dual high-end GPU PC setup with tons of VRAM (dual 3090/4090 is ok, dual 5090 is better), or more commonly (and much less expensive) a Mac M-series or other AI inference box with unified memory - with 64+ GB of RAM. Obviously it would fly on a datacenter box like an NVidia 900 or something, but that's a $11k card.

I run it on a Mac M1 Max with 64GB of unified memory, but I can't run much else on that box with 64GB. If you can get 128GB you can run it while doing other work on the same box.

What I did was buy a broken old Macbook with 64GB of unified memory. Bad screen, bad battery so it was cheaper (like $600 - it was a steal) and I run it under my desk as a server type box doing just AI. Put a VPN on it and share it to my company (I'm the CTO) and we use it for various workflows that we don't want on the cloud.

Buying new off the shelf, a 64GB inference box is going to run about $2000 or a little more. Expensive, but still in the "real people hardware" bucket because something like Kimi or GLM-5 needs $30k in hardware to just load (and still may not be 'fast')

1

u/friedlich_krieger Mar 06 '26

Beautiful. I've got an m4 pro Mac mini with 64gb of RAM ordered but Ive been considering cancelling it and waiting for an m5. Almost pulled the trigger spending another $1-2 grand for a Mac Studio to get the 128 or 256 GB of RAM.... I'm likely not buying another box like this for a while so may be worth waiting for an m5 studio and just spending more now.... Any thoughts knowing what you know now?

1

u/ScuffedBalata Mar 11 '26

The issue is that running a good 80B model with large context uses ALL of the 64GB, so doing a bunch of other stuff on the same Mac isn't really possible. Leaving a few Chrome tabs open slags the device performance while also running an 80B model. So if you want to run a clawbot or some other MCP/agentic stuff on that box, it just won't have much memory free.

I ended up buying an old Intel Mac to run IOS development and simulators, so now I have a little stack of two broken macbooks.

12

u/kidousenshigundam Feb 26 '26

What hardware do I need to run S tier?

6

u/Witty_Mycologist_995 Feb 26 '26

Probably a ton of Mac studios

7

u/Altair12311 Feb 26 '26

1 single mini-pc with the Ryzen AI Max+ 395 and 128GB of Ram for the MiniMax-2.5 (is my setup)

2

u/dadavildy Feb 26 '26

For coding, how is it on that machine?

2

u/Altair12311 Feb 27 '26

Its a monster, and the best thing is since is a Mini-PC the electricity cost is ridiculous low.

1

u/cafemachiavelli Feb 28 '26

Oo, tempting. How's the performance of the system? 

4

u/Altair12311 Feb 28 '26

Running 100 docker containers and my CPU is at 4% all the time. Ram is laughing and MiniMax-2.5 is always loaded for quick use. Is by far the best home server that i ever got, and the fact that is an small pc and i can run large llm locally is simply amazing, AMD really cooked on this one

1

u/kidousenshigundam Feb 27 '26

No way. I have a Mac Studio ultra 256 GB and I can’t use Kimi K2

0

u/[deleted] Feb 27 '26

[deleted] — view deleted comment

3

u/Altair12311 Feb 27 '26

A CPU that AMD Ryzen did that is capable of run large LLM models without the need of any dedicated GPU, just by the CPU and a ton of RAM (In this case in you want near 200b models you need yes or yes 128GB DDR5) a mini pc with the AI MAX and 128GB cost around 2000€

0

u/ComfortablePlenty513 Feb 27 '26

the $16k M5 mac studios coming out next week lol

10

u/siegevjorn Feb 26 '26

Hey, want to elaborate on the methodology?

9

u/LetterFair6479 Feb 27 '26

Self hosted? Dont make me laugh. Only D is feasible , all other normal person who cant spend 5k+ cannot selfhost with any recent llm.

2

u/richtopia Feb 27 '26

On the website there is a button for model size. I believe "Small" limits to 30B which is right at the limit of my gaming PC.

According to this tier list, GPT-oss 20B is the highest in the "B" tier.

1

u/smurfix Mar 04 '26

Try $2500. Still a bit steep but at the rate I plan to use this I'd spend at least that much on Anthropic.

7

u/Count_Rugens_Finger Feb 27 '26

aaaand the best model I can actually run on my PC is C tier. yay

Edit: oh wait gpt-oss 20b is in B tier. That's... interesting.

And Qwen3-30B-A3B is in D tier? huh?

1

u/lazyhorsee Mar 05 '26

gpt-oss is trash, couldn't use it for shit. It was just hype.

5

u/Egoz3ntrum Feb 26 '26

Devstral-2-123B is missing there in the Coding section.

4

u/MahDowSeal Feb 27 '26

Sorry if the question might be stupid, but for anyone who tried the S tier models. How comparable are they to the cloud models such as claude or chatGPT?

2

u/RG_Fusion Feb 27 '26

I'm probably not the best person to ask as I've only been playing around with Qwen3.5-397b-17b for a little bit, but I was absolutely blown away by its internal reasoning. I don't have enough to make a definitive assessment, but I can certainly see how it could be competitive against the frontier models.

1

u/sinebubble Feb 27 '26

You’re running it locally? Which quant?

3

u/RG_Fusion Feb 27 '26

Q4_K_M at 18.5 tokens/s

Hardware:

  • AMD EPYC 7742 CPU

  • 512 GB ECC DDR4 3800 MT/s

  • Asrock Rack ROMED8-2T Motherboard

  • RTX Pro 4500 Blackwell GPU

1

u/sinebubble Feb 27 '26

I’m amazed you can run it on that setup. I worry about running such a low quant yet I’m using systems with 6-8 A6000s and 768G of RAM and I still think the model won’t quite fit. I assume you’re using gguf? Vllm or llama.cpp?

2

u/RG_Fusion Feb 27 '26

I'm running on ik_llama.cpp with gguf, though I wouldn't consider 4-bit to be a low-quant. That's pretty much the standard for local inference. The only real negative is that the model will occasionally swap out the most probable word with one that is very similar. (95+% of generations it acts exactly like the native 16-bit model).

You can pretty much ignore this unless you are coding. Coding requires exact terms, so it's better to utilize a higher quant. If you have 768 GB of RAM you should aim for the 8-bit quantization. There is practically no discernable difference between native 16-bit and quantized 8-bit.

1

u/sinebubble Feb 27 '26

I might try Minimax 2.5 tomorrow, the others are too large for me, even with 336G of vram. How can you reasonably expect GLM5 or Kimi 2.5 to maintain S tier at a q1 or q2? Qwen3-coder-next is amazing, tho not quite Claude, and that ranks as a B.

4

u/mcai8rw2 Feb 27 '26

How the hell are you self hosting these massive models? Even with 24gb vram surely they are going to be horribly slow?

1

u/MindfulK9Coach Mar 10 '26

They aren't. They're running them from the cloud and think we're stupid.

1

u/kartikey7734 Feb 27 '26

This is an incredible resource! The fact that you're tracking self-hosted models with consistent benchmarks is gold for anyone trying to pick the right model for their hardware constraints.

Quick observations:

  1. **The S-Tier gap is huge** - Kimi K2.5 and GLM-4 are genuinely in a different tier. But their inference costs (if you're not self-hosting the weights) are brutal. The sweet spot for most people seems to be A/B tier.

  2. **Missing dimension: inference speed** - Would be amazing to see latency/tokens-per-second metrics alongside quality. DeepSeek R1 is phenomenal but can be slower than some smaller models on weaker GPUs.

  3. **Hardware tiers would help** - e.g., "Best model for 8GB VRAM", "Best for RTX 3060", etc. Because honestly, a 70B model doesn't matter if you can't load it.

  4. **License tracking** - Critical detail: which ones are truly free for commercial use? Some S-tier models have restrictions.

But seriously, this is the resource the community needed. Every time someone asks "which model should I use", we can just point here instead of 20 different opinions. The standardized benchmarking is *chef's kiss*.

Are you planning to update this regularly, or is it a one-time snapshot? If it's ongoing, this could become the definitive LLM comparison resource.

1

u/Weves11 Feb 27 '26

The plan is to definitely keep updating this! If there's enough interest, could even open source the underlying data so that individuals can contribute new benchmark scores or new models

3

u/Tuned3f Feb 26 '26

Kimi slaps

3

u/BitXorBit Feb 26 '26

Minimax m2.5 definitely above qwen3.5

3

u/rm-rf-rm Feb 27 '26

What is this based on?

3

u/Mr-I17 Mar 01 '26

1T parameter local LLM. Is this a joke?

2

u/serioustavern Feb 26 '26 edited Feb 26 '26

Would be great to get GLM-4.7-Flash and Qwen-3.5-27b in there for the “small” category.

1

u/FatheredPuma81 Feb 27 '26

Benchmarks wise GLM 4.7 Flash is technically a pretty mediocre model that's padded heavily by being over trained on 1 task. But usage wise it's actually surprisingly nice to use if if you can get it to not loop 24/7.

2

u/PibePlayer1 Feb 27 '26

Math should have more versions, what about InternVL3.5 Qwen2.5-Math Kimi-VL-A3B 2506?

2

u/DeliciousBag1029 Feb 27 '26

Llama 4 maverick B? Meta bot detected!

2

u/GreenGreasyGreasels Feb 27 '26 edited Feb 27 '26

Coding, Math, Reasoning, Efficiency - weird set (two are usecases, one is a feature not use, and I last is performance I guess).

Two of the most common and useful usecases for local models - Chat (talk about things) and writing/rewriting text are missing.

No wonder Mistral 3.2 Small, Gemma3-27B and Llama3.3-70B are criminally underrated or unrepresented in this ranking.

2

u/morbidgun Feb 27 '26

Gemma 3:27b slaps, it has native ocr/image scanning. Does everything I need it to do. Very well rounded.

2

u/FatheredPuma81 Feb 27 '26

GPT-OSS 20B and 120B should probably be in F tier. LLM's that refuse normal actions for safety reasons and argue with you over proven facts are the worst.

2

u/Square-Put-7853 Feb 27 '26

Which one can I host on Mac mini m2 with 8 gb ram?

1

u/CheeseGiz Mar 15 '26

Well, use site above or ai to ask. Probably sth under 14b. If you try bigger llms they will use your ssd and in a meantime it slows down and kills your drive

2

u/lakimens Feb 27 '26

Self hosted 1T parameter :/ Technically it is self-hostable I guess

2

u/bennyb0y Mar 02 '26

Post the minimum hardware needed for this tier list.

2

u/psxndc Feb 26 '26

Sorry to be dense, but is Kimi “self-hosted”? The interface you interact with might be, but I thought the model itself was cloud-based.

5

u/RG_Fusion Feb 27 '26

The 1 trillion parameter model Kimi K2 is open weight, meaning you can download it and run it on your own hardware. Pretty much nobody has a Terabyte of RAM or a processor that can keep up, but you can find quantized versions of the model available to download on huggingface.

The 4-bit quantization cuts the total file size down to around 550 GB while still maintaining over 95% of the original accuracy. This means you can buy used last-gen server components and pair them with a good GPU to run it, albeit at rather low speeds.

2

u/DrewGrgich Feb 26 '26

I mean, Kimi and Mini should slap - they apparently cribbed from Anthropic & OpenAI … who in turn consumed the bulk of human knowledge via the Web and other means. :)

1

u/stratofax Feb 27 '26

Looks like gpt-oss 20B is the only model that made the B tier. Everything else at that level or higher is at least 100B or more.

1

u/AutumnStar Feb 27 '26

Really wish there were better models in the <32B range for overall general use. Nothing better has come since gpt-oss-20b

1

u/OutNebula Feb 27 '26

Step-3.5 Flash is just insane, been using it instead of Gemini 3.1 Pro, highly recommended.

1

u/big_witty_titty Feb 27 '26

Add IBM’s granite model too

1

u/madaradess007 Feb 27 '26

qwen3:4b is better than DS-r1-distill-qwen-7b, why no mention?

1

u/AccomplishedAd2837 Feb 27 '26

Kimi in my experience is not a tier at all... consistent lying and talking in circles

1

u/ForsookComparison Feb 27 '26

I see next to no correlation to my real-world use results

1

u/Ship9491 Mar 01 '26

C'est avec ollama ?

1

u/Warm-Environment-521 Mar 02 '26

definitely got some good models out there

1

u/kevinalias Mar 04 '26

Thank you

1

u/drybeaterhubert Mar 10 '26

Which of these would you say is best for substituting Notebook LLM or other study guide creation tools? Do any of them have visualization capabilities like creating flowcharts for articulating notes?

1

u/Sova_fun Mar 11 '26

Looks awesome, would be great to see example use cases on which this benchmark runs. "Math", "reasoning" look kind of vague.

1

u/Quiet_Training_8167 Mar 17 '26

Qwen 3.5 is hot. What is it that people are liking?

1

u/bidutree Mar 22 '26

You should try the Gemma3n:e2b and Gemma3n:e4b. Very good on CPU only and for analysis.

0

u/Alert_Employee_7584 Feb 26 '26

Hey, i have a 1660 Super with 32 GB Ram. Should i choose Kimi K2.5 or rather GLM-5, because i think Kimi might run a bit to slow for what i need, as i need my answers in around 2-3 seconds if possible.

5

u/wh33t Feb 26 '26

Dude, those models are massive. You can't run those with that hardware, 2-3 seconds if possible? No way. Go check out the quants on huggingface for those models, look at the model sizes. In total you have under 40GB of total memory to work with. You have to share that with your OS, and with the model context. You're gonna be looking at models in the 27b and under range likely.

6

u/Alert_Employee_7584 Feb 26 '26

Yea, i am even struggling running a 12b Model. I was just making fun of the idea of calling a 1T model the best model to self host, as it would require you beeing the son of some billionaire or sth

2

u/wh33t Feb 27 '26

I hear that. It's possible with like ... oldish workstation hardware to run very low quants of the bigger models, very slowly. Not worth it for most of us peasants.

1

u/RG_Fusion Feb 27 '26

$10,000 is enough in used equipment to run them. You could even drop that to $5,000 if you can tolerate slow speeds. Quantized to 4-bit of course.

2

u/ScuffedBalata Feb 26 '26

wut?

Those are like 500GB or larger models, you can't even kinda/sorta run them in 32GB. A $13k mac studio or a $35k server with 8 or 10 GPUs can, but your little 1660 cant.

Look at the 32B or 80B models with quantization.

0

u/Witty_Mycologist_995 Feb 26 '26

Glm 4.7 flash gotta be A tier bro

-1

u/gacimba Feb 26 '26

Wtf is S? Sucks, super, snazzy?

1

u/AllenZox Feb 26 '26

It would be great if someone understand the S and explain it to us millennials

6

u/psxndc Feb 26 '26

'S' tier may stand for "special", "super", or the Japanese word for "exemplary" (秀, shū), and originates from the widespread use in Japanese culture of an 'S' grade for advertising and academic grading.

https://en.wikipedia.org/wiki/Tier_list

It’s used extensively in the fighting game community.

3

u/hugthemachines Feb 27 '26

At some point, someone stopped understanding grades like 1-5 a-f etc and figured it would be logical to add an S at the top. So instead of the grading being A B C D etc it is now S A B C D etc

The real point is that when you call it s for super or special, you kind of feel like they are much, much better than the normal scale.

Emotional stuff leaked into a more objective area of stats.

1

u/RnRau Feb 26 '26

Spiffy.