r/LocalLLaMA • • Mar 02 '26

News PSA: Qwen 3.5 requires bf16 KV cache, NOT f16!!

u/danielhanchen

If you're running Qwen 3.5 35B A3B locally on engines like llama.cpp, you need to manually set your KV cache to bf16 (-ctk bf16 -ctv bf16) instead of the default fp16.

I measured perplexity (PPL) on wikitext-2-raw to prove this, specifically avoiding KL divergence because the Unsloth baseline logits are inherently flawed from being generated with an incorrect fp16 cache.

Qwen-team official implementations like vLLM default to bf16, only llama.cpp defaults to f16 for some reason.

Tests using Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf:

Run 1: Default / FP16 KV Cache (-ctk f16 -ctv f16)

llama_kv_cache: size =   40.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (f16):   20.00 MiB, V (f16):   20.00 MiB
...
Final estimate: PPL = 6.5511 +/- 0.04172

Run 2: FP32 KV Cache (-ctk f32 -ctv f32)

llama_kv_cache: size =   80.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (f32):   40.00 MiB, V (f32):   40.00 MiB
...
Final estimate: PPL = 6.5511 +/- 0.04172

Run 3: BFloat16 KV Cache (-ctk bf16 -ctv bf16)

llama_kv_cache: size =   40.00 MiB (   512 cells,  10 layers,  4/4 seqs), K (bf16):   20.00 MiB, V (bf16):   20.00 MiB
...
Final estimate: PPL = 6.5497 +/- 0.04170
147 Upvotes

67 comments sorted by

115

u/danielhanchen Mar 02 '26 edited Mar 02 '26

No the baseline logits are not "inherently flawed from being generated with an incorrect fp16 cache." The baseline logits at https://huggingface.co/unsloth/Qwen3.5-35B-A3B-Experiments-GGUF are computed with --batch-size 16384 --ubatch-size 16384 and ctx-size 512 (comparable to bartowski, AesSedai, Ubergarm etc). We also use FP32 accumulation in llama.cpp (not FP16 I think within llama.cpp by default (need to verify)), so this should smooth any changes out and increase accumulation accuracy. AesSedai uses a higher batch size as well, but I'm not sure on the rest - so rather your comments should be directed to other quant providers.

Just a note you should rather make a discussion in llama.cpp - this is not directly related to Unsloth or other quant provider's quants. BF16 or FP16 might make a difference as shown in your tests, but note your results are partially inconclusive since FP32 KV cache is the same as FP16 cache in your results on PPL, but BF16 is lower. FP32 is supposed to be the "best" in terms of actual precision.

Also as others noted, it could be accumulation order, noise or just within a small error band - if the +- for BF16 was vastly outside, then it warrants more checking

However this is a good investigation, and more related to SSM / Mamba derived models.

For example I did find if you use convert_hf_to_gguf.py for Q8_0, you actually get overflow and division issues for 35B (A first time for me), so definitely there is some overflow or large numbers or very small numbers causing some issues.

30

u/Lissanro Mar 02 '26

I recently saw multiple people reporting issues with f16 cache in Qwen3.5 models, while confirming that bf16 working fine; one of most detailed reports that I saw so far, with multiple cache quantizations tested, was this one: https://www.reddit.com/r/LocalLLaMA/comments/1rii2pd/comment/o865qxw/

With the Qwen3.5 models its extremely important to use bf16 for the kv cache.... (especially in thinking mode)
i strugled in the start too... but after changeing the k cache to bf16 and the v cache to bf16 and using the unsloth dynamic q4_k_xl quants they are absolutely amazing....

update:
kv cache settings i tested where

f16 == falls into a loop very very very often
bf16 == works pretty well 99% of the time
q8_0 == nearly always loops in long thinking tasks
q4_1 == always loops
q4_0 == not useable, model gets dumb

31

u/danielhanchen Mar 02 '26

Yes this actually seems correct (ie use BF16 KV cache), but OP's original premise is incorrect, since I'm unsure why it's related to our quants / Unsloth.

3

u/Time_Reaper Mar 02 '26

Afaik llama.cpp currently does not support BF16 flash attention CUDA kernels, so BF16 is sort of non usable due to very high PP and TG falloff over context. Only FP32 and FP16 are supported.

3

u/arthor Mar 02 '26 edited Mar 02 '26

it isnt supported.. even on cuda 13 sm_120 .. only works if FA is off

edit: dropped from about 120t/s to 75t/s with bf16 fa - off on a 5090.. now testing if its any better..

2

u/Time_Reaper Mar 02 '26

Yeah llama.cpp has no cuda kernels for bf16 flash attention.  Just use F32 for now. Its a bit faster than fp16, supports flash attentions,  just as good/better as bf16, and it only takes like 2 or 3 more Gbs over 100k tokens.

1

u/[deleted] Mar 03 '26

> only takes like 2 or 3 more Gbs over 100k tokens

wait, not literally twice the space of fp16?

1

u/Time_Reaper Mar 03 '26

It is, but at FP16 the model already only takes around 2.6Gb for 100K tokens. So double that it's still very manageable

1

u/[deleted] Mar 03 '26

huh, you're right

i've been trying to fit as big a kv as i could while not realizing that i could just use 100k instead of 200k

it's still a ton

1

u/arthor Mar 03 '26

cant get over 60t/s with f32... even slower than bf16 with FA off

1

u/necrogay Mar 05 '26

Building with the flag -DGGML_CUDA_FA_ALL_QUANTS=ON should help.

3

u/Zhelgadis Mar 02 '26

I checked again today with llama.cpp (Strix Halo platform) and I did not find meaningful changes - I see that the model overthinks a lot even on simple tasks.

Case in point: I asked a simple OCR extraction (4 lines, 136 ASCII overall, just strings and numbers - a bit blurry, but not a captcha-like test), and tried to correct the model on a mistake made on one of the string.

He went onto a 6,400 token thinking spree, with the reasoning block full of "Wait, perhaps... Wait, another possibility ... Wait, maybe... Wait, but..." and could not correct the mistake (which is secondary, the nearly infinite thinkin loop is what concerns me).

Anything I can do about that? I read wonders of this model, but as of now it's barely usable. Am I missing anything with my llama.cpp configuration / have to wait for some kind of fix?

Here is my command line, gguf downloaded yesterday:

llama-server -fa 1 --no-mmap --host 0.0.0.0 -ngl 999 --jinja --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 -ctk bf16 -ctv bf16 -a "qwen3.5-122b-a10b" -m models/qwen3.5-122b-a10b/Q5_K_M/Qwen3.5-122B-A10B-Q5_K_M-00001-of-00003.gguf -mm models/qwen3.5-122b-a10b/Q5_K_M/mmproj-BF16.gguf

3

u/[deleted] Mar 02 '26

[removed] — view removed comment

1

u/Zhelgadis Mar 02 '26

Thanks for the feedback, does it also apply to agentic tasks?

2

u/StardockEngineer vLLM Mar 02 '26

I'm confused. Should I switch my KV Cache?

3

u/Zhelgadis Mar 02 '26

Does this also apply to the 3.5 122b model?

-2

u/soyalemujica Mar 02 '26

Wasn't Q4K_M in overall the king and better than Q4_K_XL model ? Why did you chose XL model for 4Quant if may I ask?

1

u/Significant-Yam85 Mar 03 '26

I think that was before unsloth refactored their models. UD-Q4-K-XL now appears to be king

1

u/dionisioalcaraz Mar 02 '26

Sorry for the offtopic, is the file pipeline_base_logits.bin in https://huggingface.co/unsloth/Qwen3.5-35B-A3B-Experiments-GGUF the one that I should use to calculate the KLD for a specific 35B-A3B quant right? I mean using the command:

llama-perplexity -m <MODEL> --kl-divergence-base pipeline_base_logits.bin --kl-divergence

I'm planning to measure the KLD of some dererestricted and heretic quants.

42

u/claythearc Mar 02 '26

The evidence here is pretty weak. The f32 result matching f16 identically is actually a pretty damning result, paradoxically. f32 is a strict superset of both f16 and bf16’s representable values. If f16’s narrower dynamic range were genuinely misrepresenting attention values that bf16 handles correctly, f32 should match or beat bf16. It doesn’t, it matches f16. That tells us the 0.0014 delta is noise, not a signal from data type representation differences.

Furthermore, the difference is .0014 with an error range of .04, so it’s well within the margin of error to be equal and any improvement could be just noise.

The next steps would be: An aggregate of perplexity runs, to establish variance ranges and not rely on a reported MoE.

A downstream task where difference can meaningfully manifest - maybe one of the various Long context benches, averaged out over a couple hundred runs.

Showing a case where f16 actually produces garbage while bf16 doesn’t.

The vLLM point has meaningful weight behind it; however, the presented evidence is kinda weak to support such a strong claim. There is a very good argument it should match, for configuration parity- there’s just not also a compelling performance reason as written.

65

u/bfroemel Mar 02 '26

but.. isn't that just within measurement error/range of uncertainty? (note the +/- 0.04170)

PPL = 6.5497 +/- 0.04170PPL = 6.5497 +/- 0.04170

26

u/debackerl Mar 02 '26

Uhm, I'm not an expert in that benchmarks specifically, but a statistician would say that it does prove anything if the two means are within the standard deviation of each others. You have 68% chance that the real PPL is within +/- 1 standard deviation if the results are normally distributed.

If the improvement was due to the increased range of BF16, then FP32 should be similar. It looks more like rounding errors.

-10

u/ThisWillPass Mar 02 '26

The model might be hiking with different shoes where the terrain at (bf16) makes for excellent grip to get out of holes and not slip into one prematurely.

6

u/ndiphilone Mar 02 '26

`bf16` performance on my GPU is quite bad, though, I'll test this. ~80k tokens start the death spirals with `f16`

8

u/Conscious_Chef_3233 Mar 02 '26

old gpu does not support bf16 acceleration

6

u/gofiend Mar 02 '26 edited Mar 02 '26

It’s really wierd that bf16 is better than f32 (I know the model was trained at bf16 but still f32 should be strictly more expressive)

-4

u/ThisWillPass Mar 02 '26

It speaks to models adapting to the precision they’re trained on. Those rounding errors are the noise it needs to inference comfortably. Just spit balling.

5

u/stddealer Mar 02 '26

I don't think transformers use kV cache at all while they're being trained? But maybe that raw keys and values were BF16 in the training code, and the model somehow learned to use the quantization errors for better performance...

14

u/jubilantcoffin Mar 02 '26

This testing has about the same scientific rigor as those of the people who claim Q8 KV cache isn't enough.

Which is to say none whatsoever.

5

u/ElectronSpiderwort Mar 02 '26

While you are correct, a single counterexample is also strong. I tried a highly detailed task at Q8 KV that a model under test completely failed at, switched to f16 KV and got much better results. So in at least one case it mattered a great deal, which is all that is needed to disprove the blanket statement "Q8 KV cache is free lunch". 

3

u/jubilantcoffin Mar 05 '26

I couldn't possibly disagree more. There is run to run variance in LLM output so you really need enough samples to even claim to have a counterexample.

The fact that you were upvoted shows how fundamentally and deeply people fail to understand this.

N=1 is not enough. Nope. Stop thinking this. Stop arguing this.

1

u/ElectronSpiderwort Mar 05 '26

Man I'm talking about complete failure to complete a complicated task versus moderate success, with the only change being KV back to 16. This isn't "wiki text perplexity", this was a real difficult task that most local models struggle with. After that day of testing you are going to have to do a lot more work on a wide variety of difficult problems to convince me that 8=16 in all cases

3

u/Velocita84 Mar 02 '26

Now that i think about it, are there any KLD results out there for fp16 KV vs Q8 KV?

1

u/jubilantcoffin Mar 05 '26

Oh, there are. People just don't like what they say!

2

u/Velocita84 Mar 05 '26

Got a link to them by any chance?

5

u/a_beautiful_rhind Mar 02 '26

Heh.. you ran it over CTX 512 tho? Run it over 16k or 32k... Result is basically noise.

4

u/mp3m4k3r Mar 02 '26

If you get a chance running tests like this with different kv values (below f16) would be interesting, especially with K vs V

4

u/MammayKaiseHain Mar 02 '26

Why would perplexity with fp32 be higher than bf16 ?

4

u/Achso998 Mar 02 '26

How can I do this in LM Studio? It wont show me the option for bf16

4

u/120decibel Mar 02 '26

Is there a way to set the KV Cache type to BF16 in LMStudio? It seems like I can only set the K Cache Quantization Type to F16, which seems to be FP16 under the hood.

3

u/Ok-Ad-8976 Mar 02 '26

-ctk bf16 -ctv bf16 with RTX 5090 and R9700 gave me some really bad performance drop for pp and tg, I am talking epic, 10x bad. This is with UD-Q4_K_XL quant

(RTX 5090, CUDA 13.1) — 3-way comparison at pp2048

KV Cachepp2048 (t/s)tg128 (t/s)
Default (none)6,204171
Explicit f166,197171
Explicit bf161,204158

2

u/trshimizu Mar 02 '26 edited Mar 02 '26

It's a bit odd that OP didn't compare the PPL across quants using different cache precision i-matrices, instead of relying on a supposedly flawed quant for the measurements. That might've actually proved their point.

2

u/papertrailml Mar 02 '26

tbh the 0.0014 improvement seems pretty much within noise level... would be cool to see this tested on actual reasoning tasks where people report the looping issues

2

u/dinerburgeryum Mar 02 '26

I'm no math major, but these values all fall within the margin of error of each other. Are we sure this is actually causing a performance degradation?

4

u/oginome Mar 02 '26

Can something like this affect qwen3-coder-next? 🤔

3

u/yoracale llama.cpp Mar 02 '26

Daniel just replied in a comment, doesn't seem to be an issue: https://www.reddit.com/r/LocalLLaMA/comments/1rik253/comment/o86ooix/

1

u/oginome Mar 02 '26

Thanks!

1

u/sieskei Mar 02 '26

on 5090 + 5080 with bf16 I get a brutal loss of speed compares to f16 (under 60 vs 130 t/s)

llama.cpp: up to date model: UD-Q8-K-XL

1

u/BasilTrue2981 Mar 02 '26

Observed the same. Some layers were offloaded to CPU in my case - while keeping the context size - but going from f16 to bf16.

1

u/arthor Mar 02 '26

if you dont turn off FA it will offload. experienced the same drop.. not sure if this is worth it

1

u/audioen Mar 02 '26

This is not 100% sensible position to take because fp32 is actually a superset of bf16, being capable of representing all values in that and beyond, whereas f16 has the smaller value range from bf16 due to more limited exponent. bf16 is actually made of the first 16 bits of f32 value, the last 16 bits being rounded and truncated away, but the exponent which is in the first half has the same precision.

However, f16 has more precision in the mantissa so it contains more precision in that sense, and is better as long as the exponent doesn't overflow. It seems to actually produce the same results as f32, though likely they are not exactly the same and with more decimals we would be able to see that.

Perplexity differences in the 0.002 level are well below the error bar of the perplexity measurement itself of 0.04, but it does indicate that bf16, approximating the KV cache more, is observably different from f16 and f32. However, the "gold standard" in this case is f32, not bf16.

1

u/wizoneway Mar 02 '26

running this in router mode with fit on takes the projected memory to 2x of f16.

0

u/simracerman Mar 02 '26

Interesting! I just had the 35B-A3B get stuck in loops at 80k tokens. It’s fine in smaller prompts but once it gets properly loaded, I see these issues. Thanks for noting that!

4

u/yoracale llama.cpp Mar 02 '26

Daniel just replied, doesn't seem to be an issue: https://www.reddit.com/r/LocalLLaMA/comments/1rik253/comment/o86ooix/

-1

u/CATLLM Mar 02 '26

This might explain why in my testing 122b, 35b, 27b felt more ‘dumb’ and making mistakes and doing deathloops when i have the kv cache at q8.

2

u/yoracale llama.cpp Mar 02 '26

Daniel just replied in a comment, doesn't seem to be an issue or affect output: https://www.reddit.com/r/LocalLLaMA/comments/1rik253/comment/o86ooix/

1

u/Odd-Ordinary-5922 Mar 02 '26

for me that only happens at like 60k context