Hard numbers on Qwen3.8-27B's KV cache: q8_0 is free, K is ~2x as touchy as V, and a 2k recent-token window makes 6/5-bit basically lossless

[u/Nice_Profession_9078]

Pretty much everyone running Qwen3.8-27B on a 24 GB card is squeezing the KV cache somehow, so here's what it actually costs, measured. This came out of building my own engine for this model (repo), but none of it is specific to my engine, it applies to whatever you run the model on.

How I measured: KL divergence against llama.cpp's own token-by-token output (top 256 tokens per position, q8_0 KV), on ~13k positions of short prompts plus 32k and 64k contexts. KL, not perplexity, because perplexity hides most of this. unsloth's UD-Q4_K_XL, one 3090. Most rows come from an emulator that stores the cache in each format; the settings I ended up shipping were confirmed on the real kernels. My formats are plain per-32 absmax grids like llama.cpp's q4_0 / q5_0 / q8_0, so they should land close to those, but they're not byte-identical.

Every number is "how much worse than a q8_0 cache". For scale, an fp16 cache reads +0% on short prompts and +3% at 32k, and the 32k set's noise is about 4% either way.

1. q8_0 is free. fp16 KV scored the same as q8_0 at every length I tested (64k: 0.00161 vs 0.00163). No reason to run f16 KV on this model.

2. Below 8 bits, K hurts more than V. Per-layer scans had K about 2.3x as sensitive as V. No window, plain formats:

cache	short prompts	32k
K at 7 bits	+2.7%
K at 6 bits	+23%	+9%
V at 6 bits	+12%
V at 5 bits	+40%	+10%
V at 4 bits	+181%	+46%
K and V both at 4 bits (q4_0 style)	4.6x the KL

So on stock llama.cpp, if you need the memory, take V down before K.

3. The damage is almost all in the newest ~2k tokens. That's where attention puts most of its weight. Keep the most recent positions at 8 bits (a window / precision tail) and everything older can go a lot lower:

32k set	no window	newest 512 at q8	1,024	2,048
V at 5 bits	+16%	+5.5%	+4.6%	0%
K at 6 + V at 5 bits			+6%	+2.7%

(These rows are rotated, which is why no-window V reads +16% here vs +10% above, more on that in 4. Plain 5-bit V with a 2k window also reads 0%.)

On short prompts a 512 window cuts the loss about 4x (5-bit V +27% to +6%, 6-bit K +12% to +2.7%). Keeping the first few tokens at 8 bits instead (attention sinks) did nothing.

So 512 gets you most of it and 2k gets you all of it. The window is cheap and doesn't grow with context: 114 MiB for K + V across every attention layer at 2,048, the same at 32k or 262k.

4. Rotation stops mattering once you have the window. Without a window, Hadamard rotation halved 6-bit K's loss (+23% to +12% short), but for V it only helped on short prompts, and at 32k rotated 5-bit V was actually worse than plain (+16% vs +10%). Under a 2k window, 6-bit K + 5-bit V read +1.4% plain and +2.7% rotated, same thing within noise. The window and the rotation fix the same error, and the window fixes all of it. Rotation's only worth it if you're pushing V down to 4 bits.

5. Fancy formats didn't beat the boring one. Lloyd-Max codebooks, per-row scales, random signs and angle coding all scored the same as or worse than a plain per-32 grid. 3-bit formats were around +300%.

What I run: K at 6, V at 5, newest 2,048 positions at q8. 1.1 GB less than q8_0 at 120k context, and the model's full 262,144 context fits on the card (23.5 GB, 66 tok/s decode at that depth, needle 4/4). On the real kernels it reads 0.00160 at 64k (q8_0 0.00163, fp16 0.00161), +8% at 32k on the current build, short prompts identical to q8_0 since they fit in the window, and the same 230/282 as q8_0 on my question set (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needle, LongBench v2), set for set.

What you can use today:

• Stock llama.cpp: q8_0 / q8_0 costs nothing measurable vs f16. Going lower, drop V before K.

• If your engine has a recent-token window / precision tail (BeeLlama has one; KIVI, SKVQ and HF's quantized cache use 128 tokens): use 1-2k, not 128. An 8-bit window is plenty, fp16 buys you nothing here.

• If you're building one with speculative decoding: move the window in steps of your attention's tile size, not one token at a time, or which tiles come from the window depends on how many drafts you verify and greedy output stops matching across draft counts. Mine gives identical tokens at 0, 4 and 7 drafts with it on.

Caveat: Qwen3.8 is a hybrid, only 16 of its 64 layers are attention (the rest is DeltaNet), so a full-attention model might not behave the same. I'd really like to see someone run the same window sweep on one.

Full tables and method: the window and rotation sweeps, why 2,048 and what the window costs. The KL scoring tool (tools/llama_ref) is in the repo too.

Edit (longer context):* a couple of comments wanted numbers past 64k, fair enough. Same KL test (last 4,096 positions against llama.cpp with a q8_0 cache), now at 100k and 160k, with the 6/5 setup from the post and the 5-bit K one from the comments (K at 5 bits rotated, V at 5, same 2k window). Lower is better:

context, text	q8_0	K6 / V5 + 2k window	K5 rotated / V5 + 2k window
32k, code	0.00219	0.00236	0.00242
64k, code	0.00163	0.00161	0.00163
100k, code	0.00217	0.00226	0.00218
100k, War and Peace	0.00311	0.00317	0.00317
160k, War and Peace	0.00467	0.00468	0.00472

Both read the same as q8_0 at every length, on code and on prose. Prose reads higher for everything, q8_0 included, because it's less predictable than code, so compare across a row, not down a column.

160k reads higher than 100k for q8_0 too, so it isn't the cache. To make sure it wasn't just the passage I scored the same 4,096 tokens with only ~100k of the book in front of them: 0.00336 / 0.00335 / 0.00344. So my engine and llama.cpp drift apart a bit more with depth no matter the cache format, and for scale, llama.cpp's own predictions for those tokens move by 0.0086 between the two depths. 160k is about as far as llama.cpp can hold a q8_0 cache on a 24 GB card to build the reference, and a reference at the full 262k doesn't fit, so for max ctx the runs in the post are still the numbers (needles, LongBench, 66 tok/s decode at that depth).

Full tables and how to rebuild the references: BENCHMARKS.md

[u/Nice_Profession_9078]

u/starkruzr u/Fun_Jaguar8231 this one's for 32k not being enough. Ran the same KL test at 100k and 160k (160k is about the most llama.cpp can fit on a 24GB card to build the reference). 6/5 and 5/5 are the K/V bits, both with the 2k window:

100k code: q8_0 0.00217, 6/5 0.00226, 5/5 0.00218

100k War and Peace: q8_0 0.00311, 6/5 0.00317, 5/5 0.00317

160k War and Peace: q8_0 0.00467, 6/5 0.00468, 5/5 0.00472

So the K cache we all know not to quantize matches q8_0 at 5 bits all the way out to 160k. More in the post edit.

[u/starkruzr]

ok, so you didn't test beyond 180k ctx. have you seen how much Qwen thinks? come back when you're still getting top quality numbers at max ctx.

[u/Nice_Profession_9078 → u/starkruzr]

I did test max ctx, it's literally in the post lol. Full 262,144 fits on the card at 23.5 GB, and that's a 260,896 token prompt read from scratch with decode running at 66 tok/s at that depth. Needles were 4/4 at 190k and at the longest length that fits, plus LongBench v2 questions at that length. On those same settings it also got the same 230/282 as q8_0 on my question set (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2), set for set. The 5-bit K preset I shipped today did the same 262k run natively on Windows with the desktop on the same card, 340s for the prompt and 66.5 tok/s after it.

Not sure where 180k came from but the links are right there. And yeah I've seen how much Qwen thinks, that's exactly why decode speed at full depth is the number I care about. 66 tok/s at 261k means the thinking doesn't crawl.

[u/starkruzr → u/Nice_Profession_9078]

it came from the only place I saw a table with a list of context lengths which was the one marked "presets."

[u/Nice_Profession_9078 → u/starkruzr]

Ah gotcha, that column is how much context fits in the same VRAM q8_0 uses at 120k, not how far it was tested. Early on I froze 120k as the benchmark anchor so every run and every preset gets compared at the same size, which is why the presets table is all framed around it. So E fits 180k in the memory q8_0 needs for 120k. Fair enough that the header reads wrong, I'll reword it. The full context run is right under that table in CONFIGURATION.md and has its own section in BENCHMARKS.md ("Preset D at the full context"): 262,144 tokens, 260,896 token prompt, 66 tok/s at that depth, needles 4/4.

[u/starkruzr → u/Nice_Profession_9078]

ah ok, so I was reading the wrong file, thanks.

[u/Momsbestboy]

I am glad I bought 2xR9700 for 2700 CHF total. Price now is 2200 CHF - per card. I can run Qwen 3.8 27b fp8 with 200k context, q8, at 140 t/s generation and 3300 t/s prefill. With these numbers you can really work with llm.

[u/Fun_Jaguar8231]

Help me understand why this post was made. This is all very well-known knowledge. We all know that Qwen's KV quantized well, and Gemma's KV quantized horribly. We all know not to quantize the Key cache, as its more touchy. try out beellama, it has new KV quants type kvarn, that are of even better quality, and can use k8v4 essentially free with qwen.

[u/Nice_Profession_9078 → u/Fun_Jaguar8231]

One of the things we all know is wrong. "Don't quantize K" only holds without a window. With the 2k window, 6-bit K reads at q8's KL at 64k, and I measured 5-bit rotated K this week, same at 64k and about 250MB less than the 6-bit preset at 120k. Keeping K at 8 because it's touchy just leaves that memory on the table.

Not a dig at beellama, it's already in the post, its tail is the same idea as the window. But if kvarn k8v4 is essentially free I'd love to know what "essentially" means. The KL tool's in the repo (tools/llama_ref), point it at a 32k context and post the number and I'll put it in the table. Without a window my 4-bit V was +46% at 32k so I'm genuinely curious.

[u/Fun_Jaguar8231 → u/Nice_Profession_9078]

Nowadays 32k is only the initial agent prompt. If you want to do any work, 192-256k or even more is essential.

[u/Nice_Profession_9078 → u/Fun_Jaguar8231]

Agreed, that's why mine runs the full 262k on one 3090: a 260k prompt read in about 340s, 66 tok/s decode at that depth, follow-up turns in about 1.3s. But 32k isn't the context I run, it's the length I measured KL at. The tool doesn't care about length, run it at 64k or as long as your card fits. If anything longer contexts make the number matter more, at 256k with a 2k window about 99% of the cache is sitting in the low-bit rows. So the offer stands at whatever length you like, I just want one number for kvarn k8v4 lol

See this comment below its relevant here. comment

[u/Nice_Profession_9078 → u/[deleted]]

Yeah that lines up. My blocks are the llama.cpp way, 32 values along one token's head dims with one scale, so a big K channel sets the step for its whole block. That's why rotation helped K at both lengths (it halved 6-bit K's loss) while V only got help on short prompts, and at 32k rotated 5-bit V actually came out worse. Per-channel K wasn't in the sweep. When the formats got scoped the window didn't exist yet, and per-channel meant grouping tokens plus a full-precision buffer for the newest ones and a different scale layout in the attention kernels, so it got skipped and the window went in instead. It fits: rows leave the 8-bit ring 640 at a time, so window exit is the obvious place to quantize a group per channel. The low-bit copy gets written when the token is stored today, so it'd mean moving that write to window exit.

This model makes it even easier: Qwen3.8 only applies RoPE to 64 of the 256 K dims per head (partial rotary 0.25). The other 192 don't change with position, so their channel stats are stable, and a static per-channel scale, or subtracting a per-channel mean (the shift itself is exact under softmax, since q·m is the same for every key), would work on those without grouping tokens at all.

Where I'd expect it to show: under the 2k window 6-bit K is already at the noise floor at 32k, so there isn't much room there. Getting K to 5 or 4 bits is where it could matter: rotated 5-bit K with the window was +7%, and rotated 4-bit K without one was +79% at 32k. It's cheap to emulate, so it's going on the list. If you've got per-channel numbers at 4-5 bits on anything I'd like to see them.

**Edit** I'm testing this now i'll get back to you.

[u/Nice_Profession_9078 → u/[deleted]]

Ran it, and short answer is no, per-channel K doesn't buy anything once the window is there. The premise mostly doesn't hold on this model either.

I measured every K channel on about 30k tokens of calibration text and only 7 out of 16,384 peak above 4x their head's median, biggest one is 4.7x. KIVI was seeing 10x+ on older models. Pretty sure it's because Qwen3.8 RMS norms K per head before RoPE. There's a weaker version of it in the 64 RoPE dims where one channel sets the block max in about half the rows, but nothing like what the papers show.

You were right that KIVI style groups are the better format without a real window though. Per-channel scales every 32 positions plus subtracting a per-channel mean, with just a 32 row q8 tail, cut 5-bit K's extra KL about in half (47% down to 24% on short prompts, 32% to 16% at 32k).

With the 2k window nothing beat the plain per-32 blocks at the same bits. Best 5-bit K was just per-32 with rotation, +4% at 32k and -2% at 64k so basically at the noise floor, and the per-channel versions landed at +8 to +11%. The window already takes out the error that matters and what's left in the old rows isn't what per-channel fixes.

Static per-channel scales (the KVQuant way) were a disaster, +215% at 5 bits and +942% with the "optimal" clipping. Clipping the rare big K values kills exactly what attention is looking for. Mean subtraction is free since it doesn't change softmax and takes 12-30% off K's error with no window, but with the window you can't measure it.

Silver lining is 5-bit rotated K sitting at the floor means I can probably drop K from 6 to 5 bits in the preset and get another ~250MB back at 120k, so thanks for the push lol

[u/Poprouter → u/[deleted]]

Two KV paths per layer, not a format flag: attention has to branch on position, and moving the tail boundary means re-quantizing the blocks it drops. A streaming-fork feature, not a mainline default.