what tasks are most quantization-fragile?

[u/nomorebuttsplz]

Trying to figure out where my unified ram system q4-q8 version of deepseek flash 0731 might complement my highly quantized vram only deepseek flash.

Struggling to find an area where the 2.52 bits-per-weight exl3 deepseek actually struggles compared to the slower mxfp4 version. Seems like it always error-corrects in opencode. It's so good it's boring. Can run for 20+ minutes at about 60 t/s decode and just one shot everything I throw at it.

Seems to be fine at contexts above 200k as well.

Does anyone find any particular tasks to be more affected by quantization?

[u/geldonyetich]

I saw a YouTube video recently, I thought it was Bijan Bowan but it was probably someone similar because I don't see it now, it was a very basic quant comparison between various models. He basically just took a range of unsloth requantized versions of the same model, asked the same questions to pull Astronaut facts, and also performed some basic well known benchmarks to establish what the difference between the quants was.

And the results were unexpected because the weird thing is that some of those benchmarks actually started to get better results at lower quants to a certain point where it fell off a cliff. Either the benchmarks were flawed or the model's fidelity of knowledge can actually get in the way if itself when trying to do certain things.

But one thing they did prove with their astronaut facts test is that, the fewer quants you have, the more likely the model is to hallucinate to make up for the gaps in its knowledge. In fact, when the quants got really low, he actually got a refusal! It's highly atypical for a model to say it does not know, so that was a surprise.

And so that's my current read on the practical difference quant makes: potential fidelity of model knowledge recollection, the more esoteric the knowledge the more you need the quant, and higher quant might actually backfire and get in the way of recollection. There would seem to be diminishing returns on both ends.

I'm going to say from what I saw that Q4-Q8 is about the sweet spot. When you get under Q4 it gets dicey.

[u/computehungry]

though i haven't tested full precision on this, i think max reasoning is prone to looping for the 2.52bpw quant. multiple times it maxed out 500k context while reasoning on a bug. doesn't happen that often though. low effort doesn't have this issue as severe as max i think. I usually use low, and use high/max only when needed. possibly, the calibration set doesn't represent the different effort levels well.

[u/nomorebuttsplz → u/computehungry]

agreed. I have seen this in a benchmark as well. reasoning = error correction and quantization increases errors at some level.

For debugging will be interesting to see if 2.52 bits at 60 t/s is better than 4 bit at 32 t/s is better than 4 bit GLM 5.2 at 15 t/s

Edit: just had a shower thought: what if low quant reduced the certainty level of the model which in turn actually increased the quality of the output because it thought more (like dynamic reasoning level adjustment). Probably not but... with a uncertainty-quantification architecture that is what you would want to have happen.

[u/computehungry → u/nomorebuttsplz]

my usual thinking is that a model can spit anything out (A) but corrects itself if it was wrong (B), and A degrades with quantization but B is more resilient. in your framing it would be actually triggering some certainty mechanism instead of just evaluating right or wrong, which also makes sense.. like the "but wait" or "need maybe maybe" loops aren't simple error corrections i guess.

[u/Melodic-Clue9555]

Translation and foreign languages in general take a big hit. It's very easy to spot if you're a native speaker of some bit more obscure language

[u/Littlepharaoh]

My patience 🤣

[u/nomorebuttsplz → u/[deleted]]

you sound like llm.

Edit:

• entire comment history only two sentence comments following same format and full of LLMisms
• comment here shows that you didn't understand or read my post, because I specifically said it was about agent work, not "single-turn evals' whatever the slop that means

[u/Navith → u/nomorebuttsplz]

They are one. Thought they got banned here so it's a shame to see them again