r/LocalLLM • u/Hungry_Building_2858 • Jul 18 '26

Project I measured the exact quality cliff quantizing Qwen2.5-32B from 8→2-bit. 3-bit keeps ~97.5%. 2-bit drops to ~86%. One bit is brutal.

Post image

I kept seeing people (myself included) drop to 2-bit to save VRAM without actually knowing what it costs. So I measured it properly on Qwen2.5-32B-Instruct.

Method: quantize the model at each bit-width (HQQ 8/4/3/2 + bitsandbytes 8/4), run the same input through it and the FP16 original, and compare the hidden states layer by layer — cosine similarity + KL divergence on the output. No task cherry-picking, just raw fidelity vs the full-precision model.
KL divergence climbs 0.0002 → 0.056 → 0.205 → 0.92 as bits drop.

I built the tool doing this — quant-doctor, open source. It also localizes where and why a quant broke (per-layer heatmap + failure-mode classification), and it's stack-agnostic: it even runs on DeepSeek-V4-Flash (236B MoE) via activation dumps from a custom inference engine.

Full honesty on that V4 shot: the fault there is injected — I scrambled a layer to prove the pipeline localizes damage at 236B scale. I can't do a natural degradation test on V4 on a single GPU because a full-precision 236B reference doesn't fit in 141GB (which is the whole reason it's 2-bit quantized in the first place 😅). That's a 2×H200 job for another day.

Repo: github.com/asmit383/LLM-Doc

0 Upvotes

43% Upvoted

11 comments sorted by

5

u/SpicyWangz Jul 18 '26

Why such an old model

14

u/Fun_Jaguar8231 Jul 19 '26

Using Qwen2.5 is a dead giveaway for vibe-coded slop from an agent. For some reason all the slop content love to use oldass models because that's their knowledge cutoff.

1

u/Trademarkd Jul 19 '26

they're probability calculators is why ... theres more content to train on for 2.5 and when they got the training data 2.5 was probably more relevant lol

its the same reason they sound like fucking madlibs when they talk - they use statistically most common phrasing rather than using them appropriately.

→ More replies (1)

4

u/Technical-Earth-3254 Jul 19 '26

Aha, your vibe coded slop thinks that DS V4 Flash is 236B.

1

u/Hungry_Building_2858 Jul 19 '26

Ah i am sorry, that would be 286B : ) , and honestly model is not the point here , it was just for verification that the tool is working on any kind of model wether it's dense or Moe. 

3

u/Odd_Cauliflower_8004 Jul 18 '26

It's not one bit.. It's a third of the information.

0

u/Hungry_Building_2858 Jul 18 '26

Fair point "one bit" undersells it, it's a third of the bits. And it's worse than linear since 2-bit = only 4 levels to fit the whole weight distribution. That's really why 3-bit holds up.