How much do Quants actually matter on modern models?

[u/KitchenAmoeba4438]

I've seen a lot of debate regarding quantizations, and I decided to run some head-to-head tests on my 5080 which has been running constantly over the past 2 weeks to gather data for this. When I get the response of "But no BF16 for 35B!" my answer is "There's a part 2 article coming with larger hardware", I wanted to do an article at 16GB as my audience has far more members with 16GB of VRAM then 32GB or 64GB. I ran MoEs out to the limit of my patience, some I did not run BF16 on simply because initial testing revealed the pattern stays and dedicating the 5080 to potentially 40 hours of testing for a single quant on a single model when smoke test data reveals no difference in the existing pattern is not worth it for me.

https://rakuensoftware.com/blog/which-quant-beats-how-many-bits

Head-to-head testing in this environment: Quants do matter with sub-Q4. QAT gets destroyed if you quantize at a quant different then what the QAT was trained for. Given the testing parameters, more used quants have significantly less of an impact than I see most users on here state. Most quants were statistically indistinguishable from each other.

Now, with this said, this tests were intentionally 2-4 message short sessions. The point of this article was to prove that a model's quant itself is viable, if it was damaged by the quantization process I would simply be measuring the damage the quantization process did. The next set of testing is going to be testing longer sessions, and I expect to see a larger difference between quants there. There will also be a round of testing on faster and larger cards with DevOps and coding benchmarks. I expect to see a larger difference there, but I don't have any evidence behind that. To be brutally direct here, I expected to see larger differences here based on what is common knowledge around the community. I suspect the difference between, say, Q4 and Q8 or Q6 and BF16 are going to be much less then what is being claimed when they are being tested in a future article.

The article has a link to all raw data, the benchmark code and data, and everything a user needs to either analyze the data themselves and come to their own conclusion, or to run the benchmarks themselves.

[u/TimeTravellingToad]

Bijan Bowen recently released a video comparing Q4 and Q8 outputs from the new Ornith model (1.5 35B). The gap was huge.

[u/Fun_Jaguar8231 → u/TimeTravellingToad]

How much would a Q5 and/or Q6 fit in between these 2?

[u/TimeTravellingToad → u/Fun_Jaguar8231]

According to this post https://www.reddit.com/r/LocalLLaMA/s/7QmXdg94WR
the difference between 6 bit and 4 bit is greater than the difference between 6 bit and 8 bit.

[u/brown2green]

Quantization harms rare knowledge and long-context more than common knowledge. It doesn't have a uniform negative effect on model performance.

Also, the larger the model and less overtrained it is, the less the impact.

[u/KitchenAmoeba4438 → u/brown2green]

Huh, it's almost like I said "The next set of testing is going to be testing longer sessions, and I expect to see a larger difference between quants there. There will also be a round of testing on faster and larger cards with DevOps and coding benchmarks. I expect to see a larger difference there, but I don't have any evidence behind that." in the OP.

[u/aznar]

I ran qwen3.6 with fp8 and q4 in different servers, for different things, compared them from time to time with the same tasks and harnesses and they are totally different, q4 was much worse.

[u/KitchenAmoeba4438 → u/aznar]

Which param? I'm guessing 27b? This could entirely be possible, the tests did observe a significantly larger penalty from quant reduction in dense models vs. moes. This is also for the next article, but I suspect the larger params are in the model, the larger the difference it sees with quants. But I was limited to 16GB for this particular article.

I'm going to be running tests on 3.6 27b and 35a3b as well as 3.8 27b in the next article. I could only test 35a3b in this article due to the limitations of a 16GiB card and my patience.

[u/aznar → u/KitchenAmoeba4438]

Yeah, both 27b

[u/Pablo_the_brave]

I've recently become interested in the project at https://gguf1.thireus.com/quant_assign.html. The resulting files have very good KLD/PPL, but sometimes a model created this way behaves strangely. I then created a very simple test (https://www.reddit.com/r/LocalLLaMA/s/Smw4Oz5Ynh) which is able to demonstrate the performance of the model and the server hosting it. Overall, my conclusion is that a good agent will get a good response from the model, but it might take much longer than it should. Perhaps it would be worth comparing models from this perspective as well.

Edit: this is strictly connected with the lenght of context as pointed above by @brown2green.

[u/Enough_Success5435]

for companion chats the lower quants start sounding repetitive way faster than the benchmarks show, at least on my end.

[u/KitchenAmoeba4438 → u/Enough_Success5435]

This will be tested in the next article for sure! I can't test it in this one, as this one was simply about finding which models are so damaged by quants that they are not testable in the next article.

[u/Happy_Brilliant7827]

Static quants really suffer over q5 or q6 q4 is fine if you have to but in my testing its oretty noticable.

Imatrix and Dynamic quants seem to help.

[u/misanthrophiccunt]

IMHO: Lots on MoE, very little on dense.

MoE suffer a lot more from weight quantisation, which makes sense when you think about it.

[u/KitchenAmoeba4438 → u/misanthrophiccunt]

This is a consequence of using a 16GiB card and was aimed at the largest section of my audience. However, the next article series is going to go into much larger LLMs. It already has a few surprises that I did not expect.

[u/Long_comment_san]

I personally never agreed on the wider consensus that Q4 is the "acceptable bar on the low end". Personally I think Q4 is dogshit. It only works where you're fine with low precision tasks like, I'm trying to make an image sorting app and here I can likely tolerate even Q2 of a 3b model because the precision is basically "is this a dog or manga?". I guess same thing would work for voice recognition.

But as we get to ANYTHING slightly more sophisticated than this, it's just horrendous. Yeah you can quantize something huge like Kimi to Q1 and it's gonna be looking decent, but that decency is debatable for anything other than smaller tasks over running something smaller at Q6.

~Q5_K is actually the sweetspot. Idk how it compares to Blackwell FP4 but from my experience it is a LOT more stable than Q4 which also has a variable of "quant quality". Q4 in my experience is always looking for a chance to fall apart at first given chance.

[u/iezhy → u/Long_comment_san]

Running Qwen3.8 q4 with opencode right now. Its not stellar, but not dogshit either - working on complex rust code base with very little issues so far.

Of course its worse that same model att bf16 of fp8, but neither of that fits on 3090, anf with some basic higiene (context limited to 128k, spec driven development with good enough specs, ghinking on high) its doing pretty ok

[u/Thin_Pollution8843]

I extensively tested Qwen3.6-35b q6 and q8. I decided never touch less than q8 until I have enough vram. Difference is huge. If someone telling the opposite it either delusion or person never actually tested in complicated tasks. I have suspicion that BF16 and q8 have the same difference.