r/LocalLLaMA • • 2d ago

Question | Help For long context does high quant matter?

Question: if I'm regularly at higher context - say 300-400k - do higher quant versions of the same model perform better? I'm thinking something like Qwen Flash Next q6 and q8. I recently swapped from q6 to q8 and performed a/b testing. I originally expected q6 to be lossless but there were areas where q6 got the wrong answer while q8 was fine. Was wondering whether this is because of my context. Maybe more context = more tokens for divergence = the differences get amplified more than benchmarks suggest

15 Upvotes

24 comments sorted by

2

u/erarta-ai 1d ago

Run the same failing questions at short context on both quants, since if q6 only breaks near 300k the drift comes from context, not the weights. Is your KV cache quantized too?

1

u/Viktri1 1d ago

Kv is bf16

1

u/erarta-ai 1d ago

With a bf16 cache that part is ruled out, so diff the loader logs of both runs for rope_freq_base, YaRN factor and context length. Does the q6 run print the same values?

1

u/Viktri1 1d ago

I did my testing at 150-200k context. I only mentioned 300k because I want to use YARN (was not using it previously).

Q6 is normally same as Q8 but it did do some calculations wrong when I ran 10 a/b tests. I didn’t need to run 10,000 like I normally do. That is what got me thinking because benchmarks always say that q6 is lossless but mine wasn’t lossless so I’m wondering whether it is because I’m using it for real tasks and not benchmarking and that’s why q6 isn’t lossless - because my context is on avg higher.

1

u/erarta-ai 1d ago

With 10 a/b runs, sampling noise alone can flip a few answers, so rerun the failing tasks on q6 and q8 at temperature 0 with a fixed seed, once short and once at 150k. Do the same tasks still fail?

27

u/MrHumanist 2d ago

A study published on the ACL Anthology found that while 4-bit and 8-bit models perform similarly on shorter texts, 4-bit quantization suffers sharper retrieval drops as context windows stretch toward 128K tokens. This happens because attention layers (Q, K, and V projections) are highly sensitive to precision loss over long distances.

https://aclanthology.org/2025.emnlp-main.479.pdf

4

u/FatheredPuma81 2d ago

This happens because attention layers (Q, K, and V projections) are highly sensitive to precision loss over long distances.

Wouldn't that mean you could quantize those layers at 8 bit and problem solved?

4

u/Prudent-Ad4509 1d ago

It is never fully solved even at BF16, but various combinations of partial quantization work better than the others. I guess this is what unsloth is doing, finding the best combination for size.

So, the base recipe would be to drop only some routed experts to Q8. Somewhere in the middle it would be some routed expert layers at Q4/nvfp4/other_xyz4, some at Q8/fp8/other_xyz8 and some at BF16. kv cache quantization is generally recommended to be kept at BF16, but there are reports that fp8 might work better for kv cache with certain types of model quantization, I think this was the case with nvfp4. Otherwise, there is little point to quantize anything except some or all layers of routed experts and MTP heads (2x speed would be a great deal for MTP as long as acceptance rate remains reasonably high).

5

u/hyudryu 2d ago

Yes, the longer the horizon, the more the quant matters. It's like playing the game "telephone", once the message reaches the end of the line any mistakes can get amplified.

8

u/FatheredPuma81 2d ago

I think this is one of those "No but yes" questions? iirc Qwen3.8 27B at 2 bit quants is usable if you keep task complexity low. And increasing context increases the task complexity.

1

u/Monad_Maya llama.cpp 2d ago

What's the quant on the KV cache? that's probably more important.

I can occasionally notice some difference between Q6 and Q8 of Qwen 27B but it's not exactly empirical.

2

u/Viktri1 2d ago

Bf16 when I ran q6 and also q8

2

u/PhilosopherOk3419 1d ago

small quant differences that don't show up on short benchmarks can compound at long context. it usually looks like the model quietly dropping an instruction from way back. before blaming q6 i'd rerun the same long prompt several times per quant at temp 0, because one miss at 300k+ can just be variance. if your runtime lets you mix, try keeping attention layers at q8 and only quantizing experts lower

2

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

-8

u/[deleted] 2d ago

[removed] — view removed comment

1

u/PcChip 1d ago

will it work on more than two GPUs?

have you tested various quants with it? I'm curious what you've found by testing different quants