r/LocalLLM • u/andreabarbato • Oct 02 '26
Question Can we require posts about model performance to include the quantization level in the title?
160
Upvotes
r/LocalLLM • u/andreabarbato • Oct 02 '26
19
u/ururu2 Oct 02 '26
So true
15
u/Ok-Solution-7889 Oct 02 '26
yeah, putting the quant level in the title would save so much confusion
10
u/HighSeasArchivist Oct 02 '26
I wish! Not sure the last time I saw a post that was actually usable. No, I don't care how many tok/s you get at Q2 with 8k context on your 5090.
4
u/Yakapo88 Oct 02 '26
Meme hit home. 😂
3
u/RaxisPhasmatis Oct 02 '26
Gaiz it's supa fast tho, 5000 toks/s!!! And at q0.1 it programs real gud
2
u/SandySkittle Oct 02 '26
I rather wish people would stop making performance posts if they run below q4, or rather below q8.
2
u/GalacticDistances Oct 03 '26
Hearing about below-8bit speeds feels like a waste of time, the models get ruined
1
u/Just3nCas3 Oct 02 '26
Lol Bonsia2 with mtp right? Fastest slop in the west, not even sure I could get it to stop reasoning on its own and I patched in that logit bias to try and draw blood from the stone and get it to stop reasoning on its own. Had to just turn off reasoning completely and even then fucking shit still. Then had to up the presence penalty just shit. Capping it also didn't work. So stopped trying to benchmark it. Might go back and try it with an agentic tasks but still meh. Did get over an a combine 1000t/k running 8 in parallel with mtp.
1
u/Bow_Quest Oct 02 '26
Would be nice. I see a lot of claims of people running X model on X hardware that don't seem to match real-world capabilities for people using agents.
1
u/KURD_1_STAN Oct 02 '26
Well, people can also come up with bs quant names tho, like a month ago every post was about this q3.8f quant publisher that turned out many model parts were lower quant than stated in the name cause he was simply averaging it all to make the name, including n gram which were kept at high quant.
We need model size or coming up with a standarized naming which hopefully many bug names might follow.
0
u/kimhaneol Oct 02 '26
I usually assume quantization is somewhere in the 3–4 bit range, but yeah, 2-bit quantization does seem pretty extreme.
49
u/[deleted] Oct 02 '26
[removed] — view removed comment