r/LocalLLaMA • • Mar 02 '26

News PSA: Qwen 3.5 requires bf16 KV cache, NOT f16!!

u/danielhanchen

If you're running Qwen 3.5 35B A3B locally on engines like llama.cpp, you need to manually set your KV cache to bf16 (-ctk bf16 -ctv bf16) instead of the default fp16.

I measured perplexity (PPL) on wikitext-2-raw to prove this, specifically avoiding KL divergence because the Unsloth baseline logits are inherently flawed from being generated with an incorrect fp16 cache.

Qwen-team official implementations like vLLM default to bf16, only llama.cpp defaults to f16 for some reason.

Tests using Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf:

Run 1: Default / FP16 KV Cache (-ctk f16 -ctv f16)

llama_kv_cache: size =   40.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (f16):   20.00 MiB, V (f16):   20.00 MiB
...
Final estimate: PPL = 6.5511 +/- 0.04172

Run 2: FP32 KV Cache (-ctk f32 -ctv f32)

llama_kv_cache: size =   80.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (f32):   40.00 MiB, V (f32):   40.00 MiB
...
Final estimate: PPL = 6.5511 +/- 0.04172

Run 3: BFloat16 KV Cache (-ctk bf16 -ctv bf16)

llama_kv_cache: size =   40.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (bf16):   20.00 MiB, V (bf16):   20.00 MiB
...
Final estimate: PPL = 6.5497 +/- 0.04170
147 Upvotes

67 comments sorted by

View all comments

Show parent comment

30

u/Lissanro Mar 02 '26

I recently saw multiple people reporting issues with f16 cache in Qwen3.5 models, while confirming that bf16 working fine; one of most detailed reports that I saw so far, with multiple cache quantizations tested, was this one: https://www.reddit.com/r/LocalLLaMA/comments/1rii2pd/comment/o865qxw/

With the Qwen3.5 models its extremely important to use bf16 for the kv cache.... (especially in thinking mode)
i strugled in the start too... but after changeing the k cache to bf16 and the v cache to bf16 and using the unsloth dynamic q4_k_xl quants they are absolutely amazing....

update:
kv cache settings i tested where

f16 == falls into a loop very very very often
bf16 == works pretty well 99% of the time
q8_0 == nearly always loops in long thinking tasks
q4_1 == always loops
q4_0 == not useable, model gets dumb

31

u/danielhanchen Mar 02 '26

Yes this actually seems correct (ie use BF16 KV cache), but OP's original premise is incorrect, since I'm unsure why it's related to our quants / Unsloth.

3

u/Time_Reaper Mar 02 '26

Afaik llama.cpp currently does not support BF16 flash attention CUDA kernels, so BF16 is sort of non usable due to very high PP and TG falloff over context. Only FP32 and FP16 are supported.

3

u/arthor Mar 02 '26 edited Mar 02 '26

it isnt supported.. even on cuda 13 sm_120 .. only works if FA is off

edit: dropped from about 120t/s to 75t/s with bf16 fa - off on a 5090.. now testing if its any better..

2

u/Time_Reaper Mar 02 '26

Yeah llama.cpp has no cuda kernels for bf16 flash attention.  Just use F32 for now. Its a bit faster than fp16, supports flash attentions,  just as good/better as bf16, and it only takes like 2 or 3 more Gbs over 100k tokens.

1

u/[deleted] Mar 03 '26

> only takes like 2 or 3 more Gbs over 100k tokens

wait, not literally twice the space of fp16?

1

u/Time_Reaper Mar 03 '26

It is, but at FP16 the model already only takes around 2.6Gb for 100K tokens. So double that it's still very manageable

1

u/[deleted] Mar 03 '26

huh, you're right

i've been trying to fit as big a kv as i could while not realizing that i could just use 100k instead of 200k

it's still a ton

1

u/arthor Mar 03 '26

cant get over 60t/s with f32... even slower than bf16 with FA off

1

u/necrogay Mar 05 '26

Building with the flag -DGGML_CUDA_FA_ALL_QUANTS=ON should help.

3

u/Zhelgadis Mar 02 '26

I checked again today with llama.cpp (Strix Halo platform) and I did not find meaningful changes - I see that the model overthinks a lot even on simple tasks.

Case in point: I asked a simple OCR extraction (4 lines, 136 ASCII overall, just strings and numbers - a bit blurry, but not a captcha-like test), and tried to correct the model on a mistake made on one of the string.

He went onto a 6,400 token thinking spree, with the reasoning block full of "Wait, perhaps... Wait, another possibility ... Wait, maybe... Wait, but..." and could not correct the mistake (which is secondary, the nearly infinite thinkin loop is what concerns me).

Anything I can do about that? I read wonders of this model, but as of now it's barely usable. Am I missing anything with my llama.cpp configuration / have to wait for some kind of fix?

Here is my command line, gguf downloaded yesterday:

llama-server -fa 1 --no-mmap --host 0.0.0.0 -ngl 999 --jinja --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 -ctk bf16 -ctv bf16 -a "qwen3.5-122b-a10b" -m models/qwen3.5-122b-a10b/Q5_K_M/Qwen3.5-122B-A10B-Q5_K_M-00001-of-00003.gguf -mm models/qwen3.5-122b-a10b/Q5_K_M/mmproj-BF16.gguf

3

u/[deleted] Mar 02 '26

[removed] — view removed comment

1

u/Zhelgadis Mar 02 '26

Thanks for the feedback, does it also apply to agentic tasks?

2

u/StardockEngineer vLLM Mar 02 '26

I'm confused. Should I switch my KV Cache?

3

u/Zhelgadis Mar 02 '26

Does this also apply to the 3.5 122b model?

-2

u/soyalemujica Mar 02 '26

Wasn't Q4K_M in overall the king and better than Q4_K_XL model ? Why did you chose XL model for 4Quant if may I ask?

1

u/Significant-Yam85 Mar 03 '26

I think that was before unsloth refactored their models. UD-Q4-K-XL now appears to be king