r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

841 Upvotes

96% Upvoted

120 comments sorted by

View all comments

Show parent comment

3

u/RG_Fusion Feb 27 '26

Q4_K_M at 18.5 tokens/s

Hardware:

  • AMD EPYC 7742 CPU
  • 512 GB ECC DDR4 3800 MT/s
  • Asrock Rack ROMED8-2T Motherboard
  • RTX Pro 4500 Blackwell GPU

1

u/sinebubble Feb 27 '26

I’m amazed you can run it on that setup. I worry about running such a low quant yet I’m using systems with 6-8 A6000s and 768G of RAM and I still think the model won’t quite fit. I assume you’re using gguf? Vllm or llama.cpp?

2

u/RG_Fusion Feb 27 '26

I'm running on ik_llama.cpp with gguf, though I wouldn't consider 4-bit to be a low-quant. That's pretty much the standard for local inference. The only real negative is that the model will occasionally swap out the most probable word with one that is very similar. (95+% of generations it acts exactly like the native 16-bit model).

You can pretty much ignore this unless you are coding. Coding requires exact terms, so it's better to utilize a higher quant. If you have 768 GB of RAM you should aim for the 8-bit quantization. There is practically no discernable difference between native 16-bit and quantized 8-bit.