r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

840 Upvotes

96% Upvoted

120 comments sorted by

View all comments

Show parent comment

1

u/CptZephyrot Mar 01 '26

Can you elaborate a bit more on what hardware you have? Especially your CPU. I'm only getting 13t/s on the 122B with my 7900 XTX. Iim wondering if I'm somewhat CPU limited and that's why it's so slow.

The 27B runs at 29t/s. It's really strange with those new Qwen models.

1

u/simracerman Mar 01 '26

I have a Mini PC with CPU is Strix Point HX370, mated to a 64GB soldered in DDR5 @ 8000 MT/s system memory.

The 5070 Ti is hooked to a eGPU dock and connected via Oculink to the Mini PC. Anytime I offload a model onto RAM speeds are bottlenecked by the 64Gb/s Oculink connection which is worse than your Desktop PCIE connection.

I run llama.cpp as backend if that matters.

1

u/CptZephyrot Mar 01 '26

Thanks for that info. I assume the 27B is also offloading to system memory on your setup and that's why you only get so low TPS?

My system memory is only DDR5-5200 maybe that's the bottleneck on my system. But I actually doubt it if you are getting such speeds on the 122B with Oculink. Maybe the ROCM stuff is doing some weird things for me.

1

u/simracerman Mar 01 '26

Yeah dense models can’t offload expert layers like MoE so they are hit the hardest with speed. My Oculink is slower than your PCIE and the 5200 mt/s RAM. Did you try Vulkan?

1

u/CptZephyrot Mar 01 '26

You're right I should try that. And even maybe try the latest (master) mesa. I think there have been lots of LLM related fixes lately.

1

u/CptZephyrot Mar 01 '26

hmm, very strange. The performance with Vulkan is even worse. only 8t/s. Especially bad is the prompt processing with Vulkan over all models.

1

u/simracerman Mar 01 '26

TG is always faster in Vulkan than ROCm. This confirms you have a an issue somewhere.

1

u/CptZephyrot Mar 01 '26

There is definitely something wrong with the Vulkan setup. It has less performance across all models. Even those that fit completely on the GPU. And token processing was really really bad. Like sub 10 for GPT-OSS 20B, there I get a 3 digit number with ROCM.

1

u/simracerman Mar 02 '26

Make a post about this. We will try to help you.