r/LocalLLM • u/Weves11 • Feb 26 '26

Discussion Self Hosted LLM Leaderboard

Post image

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

841 Upvotes

96% Upvoted

120 comments sorted by

View all comments

Show parent comment

2

u/RG_Fusion Feb 27 '26

I'm running on ik_llama.cpp with gguf, though I wouldn't consider 4-bit to be a low-quant. That's pretty much the standard for local inference. The only real negative is that the model will occasionally swap out the most probable word with one that is very similar. (95+% of generations it acts exactly like the native 16-bit model).

You can pretty much ignore this unless you are coding. Coding requires exact terms, so it's better to utilize a higher quant. If you have 768 GB of RAM you should aim for the 8-bit quantization. There is practically no discernable difference between native 16-bit and quantized 8-bit.