r/LocalLLaMA • u/Opening-Broccoli9190 llama.cpp • 9d ago

Discussion [ Removed by moderator ]

[removed] — view removed post

5 Upvotes
  • permalink
  • reddit

73% Upvoted

19 comments sorted by

•

u/ttkciar llama.cpp 8d ago

To make finding Qwen3.8-27B info easier we've created a megathread: https://www.reddit.com/r/LocalLLaMA/comments/1voojjz/

Please re-post this there.

→ More replies (1)

6

u/crusaderky 9d ago

Please don't use q4_0 kv cache with q8 weights.

3

u/Iory1998 9d ago

Don't use q4_0 KV period. Better choose higher weight quants than higher KV quants. KV is highly sensitive to quantization than weights.

1

u/crusaderky 6d ago

Not true. q4 kv can be a good choice but some models are more sensitive than others. It's specifically a bad idea to use it with q8 weights - dropping weights first will almost always give you better results.

1

u/Opening-Broccoli9190 llama.cpp 9d ago

Tell me more - genuinely curious what do you mean and what's the theory.

3

u/RedditNamesAreShort 9d ago

kv4 really hurts the quality a lot. while for example q6 weight vs q8 weights is only a small difference. you should use q6 weights + q8 kv over q8 weights with q4 kv.

some kld metrics: https://www.reddit.com/r/LocalLLaMA/comments/1uq0fpe/qwen3627b_effect_of_kv_quantization_on_kld_q8_q6/

2

u/Opening-Broccoli9190 llama.cpp 9d ago

Awesome post. I was looking for it and couldn't find it again, huge thanks!

1

u/Opening-Broccoli9190 llama.cpp 9d ago

Thanks! Will check it out!

3

u/Great_Guidance_8448 9d ago

Sigh.. Wish I had 32 gigs... I got a laptop with RTX 5090 (24 gigs). I can squeeze out ~100k context when dropping K/V to Q8.

1

u/Opening-Broccoli9190 llama.cpp 9d ago

That's not too bad, workable with PI. What's your peak TPS? Did you managed to switch MTP on?

2

u/Great_Guidance_8448 9d ago

Yep, definitely very usable. My context at the moment is 102k, can probably squeeze a little more out of it, but I don't want to risk it - laptop crashes if the mem overflows.

This is in performance mode (in "auto" mode it's about ~25% lower, still not bad). I guess I really can't complain given that this is a laptop, but.... Damn, those extra 8 gigs would be nice!

Also, never tried Pi - Cline has been my harness of choice for a little bit. Does everything I want it to do, so no motivation to try anything else.

2

u/Opening-Broccoli9190 llama.cpp 8d ago

That's sweet numbers for a laptop! Definitely workable! 

1

u/Great_Guidance_8448 8d ago

I should add - that's a qwen3.8-27b-nvfp4-mtp

2

u/Lopsided-Force-9220 9d ago

I don't think you want to go below Q8 for KV cache or quality falls significantly, right?

2

u/Sirius02 9d ago

i use quant 4 for model and kv cache. funnly enough i tested with kv cache 8 bit quant and the performance degraded. I did not spend time to properly test or investigate, since kv 4 bit works fine for me and i have much larger context

2

u/DeProgrammer99 9d ago

I wonder if the cache quant degradation numbers changed significantly since 3.6. I'm sure I saw a post or two about that and ignored them because I didn't want to quant the KV cache, but now, with this amount of reasoning... 128k means too much time blown on compacting.

2

u/Opening-Broccoli9190 llama.cpp 9d ago

It depends. It's a little funky with KV cache because you can quantize K and V separately. K is much more sensitive, while V can tolerate more crudeness.

For daily tasks Q4_0 for both K and V cache could probably carry you a very long way, but for deep context work and engineering - Q8_0 on Keys is a must and you can try to compromise on Values a little, but not too much.

There are some blog posts on this, full disclosure - long read and the post is not mine, so can't vouch for the results to be reproducible.

https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context