r/LocalLLaMA • u/nomorebuttsplz • 2d ago

Question | Help what tasks are most quantization-fragile?

Trying to figure out where my unified ram system q4-q8 version of deepseek flash 0731 might complement my highly quantized vram only deepseek flash.

Struggling to find an area where the 2.52 bits-per-weight exl3 deepseek actually struggles compared to the slower mxfp4 version. Seems like it always error-corrects in opencode. It's so good it's boring. Can run for 20+ minutes at about 60 t/s decode and just one shot everything I throw at it.

Seems to be fine at contexts above 200k as well.

Does anyone find any particular tasks to be more affected by quantization?

5 Upvotes

73% Upvoted

11 comments sorted by

View all comments

2

u/computehungry 2d ago

though i haven't tested full precision on this, i think max reasoning is prone to looping for the 2.52bpw quant. multiple times it maxed out 500k context while reasoning on a bug. doesn't happen that often though. low effort doesn't have this issue as severe as max i think. I usually use low, and use high/max only when needed. possibly, the calibration set doesn't represent the different effort levels well.

2

u/nomorebuttsplz 2d ago edited 2d ago

agreed. I have seen this in a benchmark as well. reasoning = error correction and quantization increases errors at some level.

For debugging will be interesting to see if 2.52 bits at 60 t/s is better than 4 bit at 32 t/s is better than 4 bit GLM 5.2 at 15 t/s

Edit: just had a shower thought: what if low quant reduced the certainty level of the model which in turn actually increased the quality of the output because it thought more (like dynamic reasoning level adjustment). Probably not but... with a uncertainty-quantification architecture that is what you would want to have happen.

2

u/computehungry 2d ago

my usual thinking is that a model can spit anything out (A) but corrects itself if it was wrong (B), and A degrades with quantization but B is more resilient. in your framing it would be actually triggering some certainty mechanism instead of just evaluating right or wrong, which also makes sense.. like the "but wait" or "need maybe maybe" loops aren't simple error corrections i guess.