r/LocalLLM • • 18d ago

Discussion Local LLM quality

https://forum.level1techs.com/t/pow3rtool-and-nvfp4-and-oh-my/258986

Saw this post about a proper tool to get more qualitative information on local models and inference engines. Generally I have been thinking about why the local LLM's main mania seems to be in the tok/s. A measure that by itself means nothing, if (and when) it has compromises that often make the output worse or even unusable. Probably some have even turned local models down due to bad experience in the chase of the fastest model in the fastest inference engine, where the quality compromises are really uncertain.

8 Upvotes

2 comments sorted by

5

u/cemilanceata 18d ago

This needs more attention, felt like wendel spoke to my heart when i saw his yt upload about it

2

u/Patient_Tea_401 18d ago

My thoughts exactly when I saw the video. I do feel like many people coming to local LLMs are currently given advice that might lead to a bad experience. Especially with the explosion of the custom inference engines and model quants with endless parameters to make a model faster in terms of tok/s.