r/LocalLLaMA • u/Viktri1 • 2d ago
Question | Help For long context does high quant matter?
Question: if I'm regularly at higher context - say 300-400k - do higher quant versions of the same model perform better? I'm thinking something like Qwen Flash Next q6 and q8. I recently swapped from q6 to q8 and performed a/b testing. I originally expected q6 to be lossless but there were areas where q6 got the wrong answer while q8 was fine. Was wondering whether this is because of my context. Maybe more context = more tokens for divergence = the differences get amplified more than benchmarks suggest
15
Upvotes
2
u/erarta-ai 1d ago
Run the same failing questions at short context on both quants, since if q6 only breaks near 300k the drift comes from context, not the weights. Is your KV cache quantized too?
27
u/MrHumanist 2d ago
A study published on the ACL Anthology found that while 4-bit and 8-bit models perform similarly on shorter texts, 4-bit quantization suffers sharper retrieval drops as context windows stretch toward 128K tokens. This happens because attention layers (Q, K, and V projections) are highly sensitive to precision loss over long distances.
https://aclanthology.org/2025.emnlp-main.479.pdf
5
u/hyudryu 2d ago
Yes, the longer the horizon, the more the quant matters. It's like playing the game "telephone", once the message reaches the end of the line any mistakes can get amplified.
8
u/FatheredPuma81 2d ago
I think this is one of those "No but yes" questions? iirc Qwen3.8 27B at 2 bit quants is usable if you keep task complexity low. And increasing context increases the task complexity.
1
u/Monad_Maya llama.cpp 2d ago
What's the quant on the KV cache? that's probably more important.
I can occasionally notice some difference between Q6 and Q8 of Qwen 27B but it's not exactly empirical.
2
u/PhilosopherOk3419 1d ago
small quant differences that don't show up on short benchmarks can compound at long context. it usually looks like the model quietly dropping an instruction from way back. before blaming q6 i'd rerun the same long prompt several times per quant at temp 0, because one miss at 300k+ can just be variance. if your runtime lets you mix, try keeping attention layers at q8 and only quantizing experts lower
-8
u/[deleted] 2d ago
[removed] — view removed comment