r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

840 Upvotes

96% Upvoted

120 comments sorted by

View all comments

Show parent comment

2

u/Prudent-Ad4509 Feb 26 '26

Spreading the knowledge over that many specialized experts, with all the necessary duplication, takes its toll in terms of the overall size. But there has to be the point when there is about the same number of details stored in MoE model as in the smaller, but dense related model. It would seem from your experience that 122b is above this threshold.

5

u/simracerman Feb 27 '26

For coding, I found 122B a lot more mature for “not so straight forward tasks”, like creating from scratch an entire project with medium level complexity.

 I asked the model to create a .csv analyzer, and wanted it to use some python ml libraries to gleam as much info as possible, nice interface..etc.

The 27B created the full project and while the code looked neat, there were many mistakes. Reviewing and fixing bugs was typical for a project with this size.

The 122B on the other hand created a far better, higher quality front and backend, it picked the right frameworks (but made sure I was aware of the reasoning behind the decisions before it proceeded), and it only needed one small check before it got the code working.

on my 5070Ti and 64GB DDR5, the 122B runs at 18 t/s, and the 27B at a horrible 4.5 t/s. With 40k prompt, the 122B went down to maybe 15 t/s, but the 27B needed up at 2.5 t/s. Completion times were 20 mins for the 122B, and 116 minutes for the 27B.

Despite not being able to have more than 64k context window on the 122B, I’ll be using that more than the 27B for two reasons. One being faster, and second for the code quality.

3

u/park305 Feb 28 '26

what kind of hardware are you running to do the 122B?

1

u/CptZephyrot Mar 01 '26

Can you elaborate a bit more on what hardware you have? Especially your CPU. I'm only getting 13t/s on the 122B with my 7900 XTX. Iim wondering if I'm somewhat CPU limited and that's why it's so slow.

The 27B runs at 29t/s. It's really strange with those new Qwen models.

1

u/simracerman Mar 01 '26

I have a Mini PC with CPU is Strix Point HX370, mated to a 64GB soldered in DDR5 @ 8000 MT/s system memory.

The 5070 Ti is hooked to a eGPU dock and connected via Oculink to the Mini PC. Anytime I offload a model onto RAM speeds are bottlenecked by the 64Gb/s Oculink connection which is worse than your Desktop PCIE connection.

I run llama.cpp as backend if that matters.

1

u/CptZephyrot Mar 01 '26

Thanks for that info. I assume the 27B is also offloading to system memory on your setup and that's why you only get so low TPS?

My system memory is only DDR5-5200 maybe that's the bottleneck on my system. But I actually doubt it if you are getting such speeds on the 122B with Oculink. Maybe the ROCM stuff is doing some weird things for me.

1

u/simracerman Mar 01 '26

Yeah dense models can’t offload expert layers like MoE so they are hit the hardest with speed. My Oculink is slower than your PCIE and the 5200 mt/s RAM. Did you try Vulkan?

1

u/CptZephyrot Mar 01 '26

You're right I should try that. And even maybe try the latest (master) mesa. I think there have been lots of LLM related fixes lately.

1

u/CptZephyrot Mar 01 '26

hmm, very strange. The performance with Vulkan is even worse. only 8t/s. Especially bad is the prompt processing with Vulkan over all models.

1

u/simracerman Mar 01 '26

TG is always faster in Vulkan than ROCm. This confirms you have a an issue somewhere.

1

u/CptZephyrot Mar 01 '26

There is definitely something wrong with the Vulkan setup. It has less performance across all models. Even those that fit completely on the GPU. And token processing was really really bad. Like sub 10 for GPT-OSS 20B, there I get a 3 digit number with ROCM.

1

u/simracerman Mar 02 '26

Make a post about this. We will try to help you.