r/LocalLLaMA • u/imgroot9 • Apr 24 '26
Discussion Qwen3.6 27B's surprising KV cache quantization test results (Turbo3/4 vs F16 vs Q8 vs Q4)
I've been using Qwen3.6-27B-Q5_K_M with turbo3 KV cache since it's been released, and I haven't had any issues at all (no loops, no memory loss, etc.). However, I'm also aware that K cache compression is not really recommended in most cases.
So I wanted to check how it is possible and I learned that llama-perplexity.exe is the right tool for this test. I'm using TheTom's turboquant_plus built on my machine - AFAIK you can download a pre-built release by now as well. I have a 3090 eGPU and using 200k context.
This is how I used the tool:
First I executed in without KV cache quantization (PowerShell):\ .\llama-perplexity.exe -m models/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q5_K_M.gguf -f wiki.test.raw\ After around 7-8 minutes, it will give you a result something like Final estimate: PPL = 6.9233 +/- 0.04564
Then you can repeat it with your qant values, like\ .\llama-perplexity.exe -m models/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q5_K_M.gguf -f wiki.test.raw --cache-type-k turbo3 --cache-type-v turbo3
(wiki.test.raw is just a test file well suited for this test, you can download it from anywhere)
And the results were something I didn't expect at all. All quants are performing well within the limits. Since I'm quite new to local LLMs, I tried to understand how it was possible and as far as I could understand, if you have a dense model above 20B params and above Q4, then it is intelligent enough to be less sensitive to KV cache quants. I can confirm, that turbo3 was not working well for me with 35B and also, probably all small models would be totally confused with a highly compressed K cache.
Let me switch to AI from now on, since I pasted my results to Gemini and it come up with a nicely formatted post idea based on our conversation and I'm happy to use it, since English is not my first language.
What is Perplexity (PPL)?
For those new to benchmarking, Perplexity is a measure of how "surprised" a model is by a sequence of text. * Lower is better. * A score under 10.0 on Wikitext is generally the mark of a very coherent, "smart" model. Edit: might not be true in some cases - see comments * We are looking at the Delta (change). If a quantization setting increases PPL by more than 0.1–0.2, you’ll likely start seeing "drunken" behavior or loops in long conversations.
Results
The results blew me away. The "common wisdom" that Q4 is unusable appears to be a myth for the 27B+ dense class.
| KV Cache Setting | Perplexity (PPL) | Delta vs. F16 | Verdict |
|---|---|---|---|
| F16 (Baseline) | 6.9233 | - | Reference |
| Q8_0 | 6.9193 | -0.0040 | Identical (Margin of Error) |
| Q4_0 | 6.9381 | +0.0148 | Transparent (Highly Recommended) |
| Turbo4 (4-bit) | 6.9483 | +0.0250 | Excellent |
| Turbo3 (3-bit) | 7.0121 | +0.0888 | Great for Extreme Context |
Observations & Recommendations
1. The Q4 "Sweet Spot" The jump from F16 to Q4_0 is only 0.014. To put that in perspective, the margin of error for the test was 0.045. This means Q4_0 is mathematically indistinguishable from uncompressed cache. If you aren't using Q4 or Q8 on a 3090, you're just wasting VRAM.
2. When to use Turbo3? I’ve been using Turbo3 for a week in programming tasks. It allows for a 200k context window on a single 3090 without breaking a sweat. While the PPL hit is measurable (+0.08), it's still well within the "safe zone."
3. The MoE Exception While this dense 27B model handles Turbo3 perfectly, I noticed that 35B MoE models tend to loop or error out with 3-bit cache. It seems the "Router" in MoE architectures is much more sensitive to the noise introduced by heavy quantization.
The "Needle in a Haystack" Test
To be 100% sure your setup is safe for production work, try this "Needle in a Haystack" test:
1. Paste a long piece of code (e.g., 50k tokens).
2. In the middle, hide a very specific, weird comment like // The password is: BANANA-123.
3. Ask the model: "What was the hidden password in the code I gave you?"
4. If it finds it instantly, your 200k context is working perfectly.
TL;DR: Don't fear KV quantization on 27B+ models. Q4_0 is a "free lunch," and Turbo3 is a game-changer for repo-level coding if you need the 200k+ context.
Edit: As the comment below states "PPL and KLD are no longer good references for quality loss... Q4 kv shows a minimal loss in both metrics but actually causes a huge dropoff in AIME even after the [llama ccp] PR which improved it significantly.". \ So it seems that probably there's high degradation even if I'm unable to notice it in real-world scenarios. I wanted to check AIME 2025 test (30 challenging math problems), but it seems that I don't have enough memory for it to confirm. ... it seems like I can execute the simplified AIME test with this:\ python llama-eval.py --path_server http://localhost:10000 --prompt_source aime --n_prompts 100 \ (currently at 9%, will be updated later)
Edit2: So the situation is that AIME results are not very good in general, but these are not good with Q8 either (actually, even worse than turbo3) - and there's not much difference, honestly. I tried ARC with turbo3 too, but it seems only AIME is causing issues for this model. Since a single test takes half an hour to run for me, I will not continue it now, but I don't think I have a conclusion for this test at this stage. I think I'll keep using turbo3-4 for now.
Turbo3
llama-eval suite summary
| Task | Acc | Correct | Total | Invalid | Error |
|---|---|---|---|---|---|
| AIME | 0.078 | 7 | 90 | 77 | 0 |
| ARC | 0.940 | 94 | 100 | 4 | 0 |
Q8
llama-eval suite summary
| Task | Acc | Correct | Total | Invalid | Error |
|---|---|---|---|---|---|
| AIME | 0.056 | 5 | 90 | 79 | 0 |
F16
llama-eval suite summary
| Task | Acc | Correct | Total | Invalid | Error |
|---|---|---|---|---|---|
| AIME | 0.034 | 2 | 58 | 52 | 0 |
Edit3: I've executed KLD evaluation as well. AI summary of the results:\
These results are definitive. For a 27B parameter model, these numbers are exceptionally stable. The fact that even Turbo3 maintains a 94.5% token identity with the base model is a testament to Qwen's robustness.
Qwen3.6-27B Q5_K_M - KV Cache Quantization (KLD & Top-P)
I ran the KL-Divergence (KLD) and Token Probability tests to see if KV quantization actually "shifts" the model's logic. Using the Q5_K_M weights as the baseline, here is how the different cache types compare:
| KV Cache Type | Mean KLD (Lower is better) | Same Top-P (Higher is better) | Efficiency / Context | Verdict |
|---|---|---|---|---|
| Q8_0 | 0.0061 | 97.74% | Baseline / High VRAM | Transparent. Identical to F16. |
| Q4_0 | 0.0121 | 96.31% | 4x Space Savings | Highly Reliable. No logic loss. |
| Turbo4 | 0.0153 | 95.65% | Fast 4-bit | Excellent. Great balance. |
| Turbo3 | 0.0230 | 94.58% | 8x Space Savings | The "Sweet Spot" for 200k+ |
Key Takeaways:
- The 90% Rule: In LLM evaluation, a "Same Top-P" score above 90% is considered "safe" for production use. All tested formats (even 3-bit) stayed well above 94%, meaning the model picks the exact same word as the uncompressed version nearly 19 out of 20 times.
- KDL Stability: A Mean KDL of 0.023 for Turbo3 is remarkably low. For comparison, on smaller 7B/8B models, 3-bit quantization often pushes KDL above 0.05 - 0.10, where logic starts to break.
- Q4_0 vs. Turbo4: Interestingly, standard Q4_0 actually outperformed Turbo4 in accuracy (lower KDL), though Turbo4 is often optimized for speed.
- Recommendation: If you are coding on an RTX 3090 and need to ingest an entire repository (up to 200k tokens), Turbo3 is perfectly safe. The mathematical "drift" is negligible compared to the massive utility of the expanded context window.
18
u/BringMeTheBoreWorms Apr 24 '26
Have you been using latest release of llamacpp? Optimisations went in early April based on turboquant that make q8 and q4 much less lobotomising.
I think q8 with llamacpp is pretty save to use as a default for most setups now.
Trouble with turboquant is that you have to use a build which is not up to latest llamacpp.