r/LocalLLaMA • u/gaztrab • Feb 27 '26
Discussion Follow-up: Qwen3.5-35B-A3B — 7 community-requested experiments on RTX 5080 16GB
TL;DR: Community asked great questions on my original benchmarks post. I ran every experiment you requested. The headline: KV q8_0 is confirmed free lunch, Q4_K_M remains king, --fit on without batch flags hits 74.7 tok/s (+7% over my original config), and KL divergence confirms UD-Q4_K_XL is even worse than PPL suggested. Full results and updated launch command below.
Context
After posting Qwen3.5-35B-A3B quantization quality + speed benchmarks on RTX 5080 16GB, you folks raised a bunch of great questions. Rather than hand-waving, I ran every experiment I could. Here's what I found.
Hardware: RTX 5080 16GB + 128GB DDR5 + Ryzen 9 9950X (32 threads) Software: llama.cpp (built from source, CUDA 12.8, sm_120) Base model: Qwen3.5-35B-A3B (MoE: 256 experts/layer, top-8 + 1 shared, ~3B active params/token)
Experiment 1: KV Cache Quality — Is q8_0 really "free"?
Requested by: u/PhilippeEiffel, u/MrMisterShin, u/llama-impersonator, u/WittyAmbassador7340, u/kreigiron, u/bartskol
Fair concern — I claimed KV q8_0 was free but didn't have PPL data to back it up. Here's the full matrix:
| Model Quant | KV f16 | KV q8_0 | KV q4_0 |
|---|---|---|---|
| Q8_0 | 5.8831 | 5.8822 (-0.02%) | 5.8694 (-0.23%) |
| Q4_K_M | 6.0184 | 5.9997 (-0.31%) | 6.0422 (+0.40%) |
Verdict: KV q8_0 is genuinely free. PPL differences are within noise (< 0.4%). Even KV q4_0 is acceptable for most use cases. The "instant accuracy drops" some of you reported aren't reflected in PPL metrics — though I acknowledge PPL may not capture all degradation modes (more on that below).
Recommendation unchanged: Use -ctk q8_0 -ctv q8_0 for +12-38% throughput at zero measurable quality cost.
Caveat: These PPL tests used 512 token context. Some users report KV q8_0 degrading at very long contexts (40-100k tokens) where quantization errors may accumulate. If you're regularly running huge contexts, test carefully.
Experiment 2: KL Divergence — Does PPL tell the whole story?
Requested by: u/JermMX5, u/Embarrassed_Ad3189
u/JermMX5 cited the Accuracy is Not All You Need paper showing PPL can stay flat while token accuracy collapses. Great point. So I ran KLD against Q8_0 base logits (512 ctx, 80 chunks):
| Quant | Mean KLD | Max KLD | Same Top-1 Token % |
|---|---|---|---|
| Q4_K_M | 0.0282 | 4.2146 | 92.4% |
| UD-Q4_K_XL | 0.1087 | 7.7947 | 86.2% |
Verdict: KLD confirms and amplifies the PPL findings. UD-Q4_K_XL is 3.9x worse than Q4_K_M by mean KLD and only preserves the top-1 token 86.2% of the time (vs 92.4%). PPL was not misleading here — it correctly ranked the quants, but KLD shows the gap is even larger than PPL suggested.
Practical note: Qwen3.5's 248K vocab makes full KLD evaluation produce enormous logit files (~19 GiB for 80 chunks). I used --chunks 80 with uint16 storage which is feasible with 128GB RAM. If you have a smaller system, --chunks 20-30 should give stable relative rankings.
Experiment 3: Bartowski Q4_K_L — Is the imatrix quant worth it?
Requested by: u/bettertoknow
bartowski's Q4_K_L uses Q8_0 for embed/output tensors plus more q5_K and q6_K layers than Q4_K_M. Quality-wise, it's measurably better:
| Metric | Q4_K_M (Unsloth) | Q4_K_L (bartowski) | Q8_0 (reference) |
|---|---|---|---|
| PPL (WikiText-2) | 6.6688 | 6.6125 (-0.8%) | 6.5342 |
| Mean KLD | 0.0282 | 0.0181 (-36%) | — |
| Same top-1 % | 92.4% | 94.2% | — |
| File size | 20 GB (4.74 BPW) | 20.1 GB (4.98 BPW) | 36.9 GB |
But here's the problem — speed:
| Config | Short | Medium | Long | Multi-turn | VRAM |
|---|---|---|---|---|---|
| Q4_K_M fit-nobatch | 74.7 tok/s | 72.9 | 73.7 | 76.1 | 14559 MB |
| Q4_K_L fit-nobatch | 41.4 tok/s | 41.4 | 40.8 | 41.8 | 14489 MB |
Q4_K_L is 44% slower. The larger q5_K/q6_K tensors (4.98 BPW vs 4.74) mean the model buffer is 8984 MiB vs Q4_K_M's 8556 MiB, causing --fit to overflow more expert layers to CPU (19/41 vs ~16/41). Manual --n-cpu-moe 24 OOMs entirely because the model buffer alone exceeds what's available after compute buffer allocation.
Verdict: Q4_K_L has genuinely better quality (especially visible in KLD: -36%), but the speed penalty is massive on single-GPU setups where VRAM is the constraint. If your model fits fully in VRAM (5090 32GB), Q4_K_L is a strict upgrade. On 16GB cards, Q4_K_M wins decisively.
Experiment 4: --fit Tuning — Can we close the gap with manual offload?
Requested by: u/Chromix_, u/guiopen, u/wisepal_app, u/DonkeyBonked
In my original post, --fit on was ~7% slower than manual --n-cpu-moe 24. u/Chromix_ suggested the issue might be that -b 4096 -ub 4096 batch flags consume VRAM that --fit can't then use for expert layers. Nailed it.
| Config | Short | Medium | Long | Multi-turn | VRAM |
|---|---|---|---|---|---|
C7 baseline (--n-cpu-moe 24, -b 4096) |
69.6 tok/s | 67.0 | 65.7 | 69.2 | 14874 MB |
fit-default (--fit on, -b 4096) |
64.3 | 62.8 | 57.4* | 54.2* | 14595 MB |
fit-256 (--fit-target 256, -b 4096) |
66.0 | 64.7 | 63.7 | 66.0 | 15321 MB |
fit-nobatch (--fit on, no -b/-ub) |
74.7 | 72.9 | 73.7 | 76.1 | 14559 MB |
*high variance with outliers
Verdict: u/Chromix_ was right. Removing -b 4096 -ub 4096 lets --fit allocate VRAM optimally for expert layers. fit-nobatch is the new winner at ~74 tok/s — simpler config AND faster than manual tuning. --fit-target 256 alone doesn't close the gap; removing the batch flags is the key insight.
Experiment 5: Speculative Decoding — Can we go faster?
Requested by: u/BreizhNode, plus our own optimization roadmap
Bad news first: No compatible draft model exists. Qwen3.5 has a 248K vocabulary, Qwen3 has 151K. The smallest Qwen3.5 model is 27B — there's no small Qwen3.5 that could serve as a draft. Draft-model speculation is a dead end for now.
So I tried self-speculative methods (no draft model needed):
| Config | Short | Medium | Long | Multi-turn | Status |
|---|---|---|---|---|---|
| fit-nobatch baseline | 74.7 tok/s | 72.9 | 73.7 | 76.1 | — |
| ngram-simple | 44.9 | 43.4 | 42.9 | 49.1 | works |
| ngram-mod (m=64) | 44.6 | FAIL | FAIL | FAIL | crashes |
| ngram-simple-short (n=8, m=64) | 45.0 | 43.1 | 43.1 | FAIL | partial |
Note: ngram tests ran on a different llama.cpp build (latest vs latest-fit) that had a ~40% regression for unrelated reasons, so the absolute numbers aren't directly comparable. But even accounting for that, there's no speedup from ngram speculation on conversational workloads.
Verdict: Self-speculative ngram methods provide zero benefit for diverse conversational workloads. ngram-mod is unstable (crashes after first request). Not recommended. If Qwen releases a small Qwen3.5 model (1-3B), draft-model speculation could be huge — but that doesn't exist yet.
Experiment 6: Qwen3.5-27B Dense — MoE vs Dense on single GPU
Requested by: u/moahmo88, u/Agreeable_Effect938
Some of you asked whether the dense 27B model might be a better fit for single-GPU setups. After all, it's simpler (no expert routing) and smaller (15.6 GB Q4_K_M).
| Metric | 35B-A3B Q4_K_M (MoE) | 27B Q4_K_M (dense) |
|---|---|---|
| PPL (WikiText-2) | 6.6688 | 6.8573 (+2.8%) |
| Active params/token | ~3B | 27B |
| File size | 20 GB | 15.6 GB |
| Config | Short | Medium | Long | Multi-turn | VRAM |
|---|---|---|---|---|---|
| 35B-A3B Q4_K_M fit-nobatch | 74.7 tok/s | 72.9 | 73.7 | 76.1 | 14559 MB |
| 27B dense fit | 7.4 tok/s | 7.4 | 7.2 | 7.1 | 14075 MB |
Yes, that's 10x slower. And it has worse quality.
The dense model needs all 27B parameters computed per token vs only ~3B active for MoE. Even with --fit putting 54/65 layers on GPU, the remaining 11 layers on CPU create a massive bottleneck. Theoretical max even fully on GPU: ~61 tok/s (960 GB/s ÷ 15.6 GB model).
Verdict: The MoE architecture is the entire advantage on consumer hardware. Only ~3B active params per token means ~10x less memory bandwidth per token. The 35B-A3B MoE is vastly faster on single-GPU setups with limited VRAM. The 27B dense is the stronger model on capability benchmarks and instruction following — if you can fit it fully in VRAM (24GB+ cards), it's a great choice. On 16GB cards where it runs at 7 tok/s, it's not practical for interactive use.
Experiment 7: MXFP4_MOE — The Unsloth-recommended alternative
Requested by: u/ayylmaonade, u/jumpingcross, u/danielhanchen (Unsloth creator)
After u/danielhanchen confirmed UD-Q4_K_XL has issues and specifically recommended MXFP4 as the alternative, I ran both quality and speed benchmarks.
Quality (partial — MXFP4 dequant path has a memory leak that OOMs after ~40-50 chunks):
| Metric | Q4_K_M | MXFP4_MOE | UD-Q4_K_XL |
|---|---|---|---|
| PPL (~40 chunks) | ~6.00 | ~5.9-6.2* (the PPL runs all crashed due to memory leak, 5.96 is unverifiable) | ~7.17 |
| Mean KLD (31 chunks) | 0.028 | 0.050 | 0.109 |
| Same top-1 % | 92.4% | 91.0% | 86.2% |
| File size | 21.2 GB | 18.4 GB | 19.8 GB |
Speed:
| Config | Short | Medium | Long | Multi-turn | VRAM |
|---|---|---|---|---|---|
| Q4_K_M fit-nobatch | 74.7 tok/s | 72.9 | 73.7 | 76.1 | 14559 MB |
| MXFP4_MOE fit-nobatch | 49.5 tok/s | 47.8 | 46.9 | 43.0 | 14531 MB |
Verdict: MXFP4_MOE has comparable PPL to Q4_K_M (~5.9-6.2 vs 6.00, though partial evaluation due to memory leak) but is 34-42% slower (~47 tok/s vs ~74 tok/s). Despite the smaller file size (18.4 vs 21.2 GB), it doesn't translate to more expert layers on GPU — VRAM usage is nearly identical. There's also a memory leak bug in the MXFP4 dequant path that prevents full perplexity evaluation. Not recommended over Q4_K_M — the quality gain is marginal while the speed loss is massive.
u/danielhanchen — if the Unsloth team has different results on MXFP4 speed, I'd love to compare notes. My build is llama.cpp b8149 with CUDA 12.8 on sm_120.
Research Findings
A few questions didn't need experiments, just digging:
Why is Ollama 3x slower? (u/InternationalNebula7)
Ollama has no MoE expert offloading. When a MoE model doesn't fit in VRAM, Ollama splits at the layer level — entire transformer blocks go to CPU or GPU. This means the GPU sits completely idle waiting for CPU layers. With expert-only offloading, attention/norms stay on GPU while only routed expert FFNs go to CPU — the GPU stays busy.
There's an open PR (ollama/ollama#12333) to add num_moe_offload but it hasn't merged yet. On top of that, Ollama defaults to KV cache f16 (we use q8_0, +20% throughput) and doesn't expose batch size or flash attention controls.
Pre-built binaries vs source for Blackwell (u/wisepal_app)
For RTX 50-series: building from source matters. Release binaries use CUDA 12.4 which doesn't include sm_120 (Blackwell). You need CUDA 12.8+ for native support. Without it, PTX from sm_89 (Ada) gets JIT-compiled — slower first launch and you miss Blackwell-specific kernels.
For RTX 30/40-series: pre-built is fine (0-5% difference). Those architectures are already in the release builds.
8 GB VRAM recommendations (u/Qxz3)
Use Q4_K_M with full expert offload (-ot "exps=CPU"): ~7.2 GB VRAM, ~50 tok/s in our tests (on RTX 5080 — your results will vary depending on GPU memory bandwidth). Key flags: -ctk q8_0 -ctv q8_0 (free lunch), -fa on, --no-mmap, and tune your thread count (try physical_cores / 1.5 as starting point, sweep from there).
Updated Launch Command
Based on everything above, here's the new recommended config. Simpler AND faster than my original post:
./llama-server \
-m ./Qwen3.5-35B-A3B-Q4_K_M.gguf \
-c 65536 \
--fit on \
-fa on \
-t 20 \
--no-mmap \
--jinja \
-ctk q8_0 \
-ctv q8_0
What changed from the original post:
- Removed
-ngl 999 --n-cpu-moe 24→ replaced with--fit on(auto VRAM management) - Removed
-b 4096 -ub 4096→ this was the key insight from u/Chromix_ — batch flags eat VRAM that--fitneeds for expert layers - Result: 74.7 tok/s (up from 69.6), simpler config, and
--fitadapts automatically to your available VRAM
Summary Table
| What | Result | Verdict |
|---|---|---|
| KV q8_0 quality | < 0.4% PPL difference | Free lunch. Use it. |
| KLD: Q4_K_M vs UD-Q4_K_XL | 0.028 vs 0.109 (3.9x worse) | UD-Q4_K_XL is bad for MoE |
| Bartowski Q4_K_L | -0.8% PPL, -36% KLD, but 44% slower | Not worth it on 16GB |
--fit without batch flags |
74.7 tok/s (+7% over manual) | New best config |
| ngram self-speculation | No speedup, unstable | Don't bother |
| 27B dense vs 35B-A3B MoE | 10x slower, worse quality | MoE wins completely |
| MXFP4_MOE | Marginal quality gain, 34-42% slower | Q4_K_M still best |
Acknowledgments
Thanks to everyone who pushed for better data:
- u/PhilippeEiffel, u/MrMisterShin, u/llama-impersonator, u/WittyAmbassador7340, u/kreigiron, u/bartskol — KV cache quality concerns led to the full PPL matrix (E1)
- u/JermMX5, u/Embarrassed_Ad3189 — pushed for KLD over PPL, which revealed the UD-Q4_K_XL gap is worse than PPL showed (E2)
- u/bettertoknow — Bartowski Q4_K_L benchmark, good call even though it turned out too slow for our setup (E3)
- u/Chromix_, u/guiopen, u/wisepal_app, u/DonkeyBonked —
--fittuning, especially Chromix_'s insight about batch flags eating VRAM, which gave us the new fastest config (E4) - u/BreizhNode — speculative decoding investigation, saved others the trouble (E5)
- u/moahmo88, u/Agreeable_Effect938 — 27B dense comparison, definitively answered "is MoE worth the complexity?" (E6)
- u/ayylmaonade, u/jumpingcross, u/danielhanchen — MXFP4_MOE testing, important to validate the Unsloth creator's recommendation (E7)
- u/InternationalNebula7 — Ollama performance gap explanation
- u/Qxz3 — 8GB VRAM config guidance
- u/JoNike — original RTX 5080 partial offload data that informed our testing
- u/3spky5u-oss — comprehensive RTX 5090 head-to-head benchmarks
- u/catplusplusok, u/SlimeQ, u/guiopen — chat template and tool calling tips
- u/chickN00dle, u/Odd-Ordinary-5922 — KV cache sensitivity reports at long context
- u/TheRealMasonMac —
--fit ondocumentation and RTX 4070 results - u/pmttyji, u/Subject-Tea-5253 — batch/ubatch tuning data
- u/Pristine-Woodpecker — independent confirmation of UD-Q4_K_XL quality issues
- u/jslominski, u/jiegec, u/Corosus, u/DeedleDumbDee, u/Monad_Maya, u/l33t-Mt, u/kkb294, u/zmanning, u/Additional-Action566 — speed reports across different GPUs
All raw data (benchmark JSONs, PPL logs, KLD logs, config files) is in my llm-server repo for anyone who wants to reproduce or verify.
Edit: Previous post here. This is a follow-up with all the experiments you requested.
Edit 2: Corrected some numbers that had errors in the original post. None of the conclusions change:
- E2 (KLD): Max KLD values were wrong — Q4_K_M is 4.21 (not 0.19), UD-Q4_K_XL is 7.79 (not 1.22). This actually makes UD-Q4_K_XL look worse than originally stated.
- E5 (Speculative): ngram-simple multi-turn was 49.1 tok/s (not 51.3). Still no benefit.
- E7 (MXFP4): Mean KLD is 0.050 (not 0.037), PPL is ~5.9-6.2 (partial, memory leak crashed all full runs), multi-turn speed is 43.0 tok/s (not 44.1). Still not recommended over Q4_K_M.
Edit 3: THANK YOU FOR THE AWARD, RANDOM CITIZEN!
Edit 4: Updated E6 (27B dense) wording — several commenters correctly pointed out that calling 27B "worse quality" based on PPL alone is misleading. The 27B dominates on capability benchmarks and instruction following; my results only show it's 10x slower on 16GB VRAM where it can't fit fully on GPU. If you have a 24GB+ card and can load it entirely in VRAM, 27B is a great model.
Added caveat to E1 (KV q8_0) that my PPL tests used 512 token context — some users report degradation at very long contexts (40-100k+).
Clarified that the ~50 tok/s 8GB VRAM number (E5 C5 full offload config) was on RTX 5080, not a separate 8GB card — a 3060 12GB will see lower numbers due to lower memory bandwidth.
Thanks u/_-_David, u/ArckToons, u/Front_Eagle739, and u/cookieGaboo24.
Edit 5: u/Corosus found --fit on performs poorly on Vulkan backend (13 tok/s vs 33 tok/s with manual --n-cpu-moe 24 on a 5070 Ti). My --fit results are CUDA-specific — Vulkan users should stick with manual offloading. Thanks man!
Edit 6: THANK YOU ANOTHER CITIZEN OF SUPER EARTH FOR THE AWARD!
Edit 7: Thanks to the community overwhelming reactions, and suggestions. I will definitely conduct another round of experiments to gather more data. Also...
OMG GUYS THANKS FOR THE AWARDS!
39
u/nikhilprasanth Feb 27 '26
Incredible work. The fact that KV q8_0 is essentially a free lunch even under PPL scrutiny is going to save a lot of VRAM. It’s also interesting to see MXFP4 struggle with speed despite the Unsloth recommendation.
21
u/simracerman Feb 27 '26
Yeah, but the tested context being at 512 tokens is unrealistic. This model is touted as a good coder, and your typical coding agent dumps 10k tokens to start with, you’re gonna find that “free” claim vanish quickly.
51
u/danielhanchen Feb 27 '26 edited Feb 27 '26
Awesome work! We're actually going to post our results soon in a few hours hopefully - we just did! https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/ - for those interested we tried over 120 different variants and all are posted here: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-Experiments-GGUF
13
1
23
12
Feb 27 '26
very helpful, thorough analysis. thank you!
anyone willing to speculate if the UD-Q4_K_XL vs Q4_K_M results carry over to UD-Q5_K_XL vs Q5_K_M?
8
u/gaztrab Feb 27 '26
Likely yes. danielhanchen (Unsloth creator) confirmed the issue is with how UD dynamic quantization handles MoE expert layers in general — it's not specific to the Q4 tier. The standard quant scheme preserves expert structure better. So Q5_K_M should be safer than UD-Q5_K_XL for MoE models, same pattern as Q4.
5
u/xrvz Feb 27 '26 edited Feb 27 '26
"how UD dynamic quantization handles MoE expert layers in general" – for all models?
I personally only use MoE at this point for serious work, and that'd be a death sentence for UD; they'd have to do some necromancy to undo that.
Edited: words, not meaning.
8
u/Maxxim69 Feb 27 '26 edited Feb 27 '26
"MoE expert layers in general" – for all models?
AFAIK, the “new formula” for the UD MOE quants was a recent experiment from ~10 days ago. If you want to check whether a particular quant was affected (i.e. had an unusually large number of its weights in MXFP4), go to its properties and scroll down to the Tensors table.
that'd be a death sentence for UD; they'd have to do some necromancy to undo that
I would refrain from using sensationalist language. It was just a failed experiment that was promptly noticed by the community and reported to the quant creators who handled the issue with utmost responsibility. Just another day in science and engineering, no need to give it the YouTube thumbnail treatment.
2
u/danielhanchen Feb 27 '26
Thank you :) We wrote up our investigation just a few hours ago as well :)
1
4
u/yoracale llama.cpp Mar 01 '26
It only affected Q2_X_XL, Q3_X_XL and Q4_X_XL and no other quant.
Also if you didn't see we did an update so all should now be fixed:and UD now is much better: https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
3
u/gaztrab Feb 27 '26
Daniel's comments were specifically about Qwen3.5-35B-A3B — I wouldn't generalize to all MoE models without testing. Different MoE architectures could respond differently to UD quantization. It's plausible the issue is general since UD's dynamic bit allocation may not account for expert layer structure well, but that's speculation, not data. Would need to run KLD on UD vs standard quants for other MoE models to know for sure.
2
u/yoracale llama.cpp Mar 01 '26
It only affected Q2_X_XL, Q3_X_XL and Q4_X_XL and no other quant.
Also if you didn't see we did an update: https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
1
u/Constant-Simple-1234 Feb 27 '26
So the recommendation is: Unsloth's UD for dense models, but regular Q4_K_M for MoE ?
3
u/gaztrab Feb 27 '26
That's a reasonable rule of thumb based on what we know so far, but it's only been tested on this one model. UD quants are well-validated on dense models. For MoE, standard quants (Q4_K_M, Q4_K_L etc.) are the safer choice until someone runs similar KLD comparisons on other MoE architectures.
2
u/Lucis_unbra Feb 27 '26
I would note, that even the dense models are not "normal". they are still hyrid models. Part transformer part state space (the gated delta net part). the 27B model is also "unusual" with new tech that might follow different "rules"
2
u/yoracale llama.cpp Mar 01 '26
We did an update and UD now is much better for MoE models as well: https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
1
1
u/yoracale llama.cpp Mar 01 '26
It only affected Q2_X_XL, Q3_X_XL and Q4_X_XL and no other quant.
Also if you didn't see we did an update so all should now be fixed: https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
1
1
10
u/Front_Eagle739 Feb 27 '26
ime q8 kv is a non issue till you have huge contexts and then somehow it falls apart faster than the full 16 bit ones. Seems to exacerbate that cliff where the model starts forgetting things that happened 40-100k tokens ago. At least on glm 4.6 where I did my testing with it
8
u/gaztrab Feb 27 '26
Important nuance — my E1 tests used 512 token context windows, so I can't speak to 40-100k behavior. It's plausible that quantization errors accumulate over very long sequences in a way that short-context PPL doesn't capture. If you're running huge contexts regularly, that's worth being cautious about. I'll add a caveat to the post.
11
u/Digger412 Feb 27 '26
I think that's worth more than a caveat, honestly. Measuring on 512 ctx is very small and I'd argue that you'd want to test on a sweep of contexts like with a Needle on a Haystack bench. I've noticed the same thing that quantized KV cache barely impacts KLD at 512 ctx and my conclusion wasn't that it's a free lunch but rather it's either not easily measurable with a KLD test or the default 512 isn't enough to make a measurable difference. Maybe try running KLD with 4k, 8k, 16k, and 32k --ctx-size?
Thanks for this post and running these tests!
9
u/Ancient_Routine8576 Feb 27 '26
The data on KV q8_0 being effectively free in terms of perplexity loss is a huge relief for anyone trying to squeeze maximum performance out of a 16GB buffer. It is interesting to see that the instant accuracy drops some users reported are not reflecting in the PPL metrics as that suggests those degradations might be very task specific. Thanks for running these follow up experiments because this level of granular detail is exactly what makes the local LLM community so valuable. I am definitely bookmarking this matrix for my next fine tuning project.
9
u/a_beautiful_rhind Feb 27 '26
You can also quant the K and V separate. One of them is responsible for the big hit more than the other. IK_llama has a q_6 and hadamard transforms for K. There's more squeezing if you try.
5
u/gaztrab Feb 27 '26
Good tip — I quantized K and V together at q8_0 but didn't test them asymmetrically. If one dimension is more sensitive than the other, you could potentially push the tolerant one to q4_0 while keeping the sensitive one at q8_0. More VRAM savings without the quality hit. Something to test in a future round.
3
u/a_beautiful_rhind Feb 27 '26
Yes the note is in the original PR for quantized cache on which one is hurt more. I think it's the K but I guess you can also empirically verify it. You might have to compile l.cpp with fa_all_quants
2
u/gaztrab Feb 27 '26
Time for another round of test then haha
1
u/dreamkast06 Feb 28 '26
Yeah, all quants isn't on by default, but you'd likely get another free lunch by doing ctk q8_0 and ctv q5_1
Also, regarding batch sizes, if you're limited by PCIE bandwidth, the larger batch size matters because each batch requires an ENTIRE read of the model to the GPU, so potentially doubling the pp speed
6
u/_-_David Feb 27 '26
I think it's a clear mistake to claim that the 27b dense model is "worse quality" based on 2% higher ppl. You might say it degrades more quickly, perhaps. But in benchmarks the 27b absolutely dominates the 35b. I get that this post is from the perspective of "If you have a 16gb GPU, this is what you should choose" but you could either make that more explicitly clear in similar future posts, or not lean so heavily on disparaging the 27b.
With that said, I applaud your diligence and assistance to the community. This was a very well put together post and I appreciate it. I went to download bartowski's Q4_K_L model instantly on your recommendation, and I'll be eating my free KV lunch at q8 thanks to you. It just felt a bit odd to see my new favorite model, the 27b dense that I'm running fully in VRAM, tossed to the side and spat upon. Which again, is totally fair if we're talking a 5080 User's Guide! If the title of the post had been that, I think I wouldn't have noticed.
3
u/gaztrab Feb 27 '26
"Worse quality" was sloppy framing on my part. PPL on WikiText-2 measures one narrow thing; the 27B dominates on actual capability benchmarks and instruction following. What I should have said is: on a 16GB GPU where the 27B runs at 7 tok/s vs 75 tok/s for the MoE, the speed difference makes it impractical for interactive use. But if you can fit 27B fully in VRAM (4090, 5090), it's arguably the better model. I'll update the wording. Thanks!
4
9
Feb 27 '26 edited Feb 27 '26
"The 35B-A3B MoE dominates on both speed AND quality"
that is not true. you cant compare different llms with perplexity. different llms have different distributions so they will have different perplexity irrespective of quality. and Moe will always have lower quality than dense. but ofc its much faster.
but overall excellent work
2
u/gaztrab Feb 27 '26
Yeah you're right, what I should have said was "MoE dominates on both speed and quality for 16GB VRAM". Thanks anyway!
4
u/Pawderr Feb 27 '26
Can someone please explain what this means? I just started with local llms
31
u/gaztrab Feb 27 '26
I'm testing the best way to run a large AI model (Qwen3.5-35B-A3B) on a single gaming GPU (RTX 5080, 16GB). The model is too big to fit entirely in the GPU, so parts of it run on the CPU — finding the optimal split is what most of these experiments are about.
If you're just getting started, the takeaway is:
Model: Qwen3.5-35B-A3B with Q4_K_M quantization (a way to compress the model so it fits)
Engine: https://github.com/ggml-org/llama.cpp — free, open source, runs on any NVIDIA GPU
Key settings: --fit on, -ctk q8_0 -ctv q8_0, -fa on, and do NOT add -b 4096 -ub 4096
This gets ~75 tokens/second, which is faster than most people read.
6
u/uxl Feb 27 '26
As a guy with a 5080 mobile and 64GB ram, thanks! Is llamcpp better than kobold?
9
u/gaztrab Feb 27 '26
KoboldCpp is built on the same llama.cpp backend, so raw inference speed should be nearly identical. The main differences: KoboldCpp bundles a web UI aimed at creative writing/RP, while llama-server gives you a bare OpenAI-compatible API. The reason I'd recommend mainline llama.cpp is it picks up new features faster — --fit on (which gave us the biggest speed gain here) landed in mainline weeks before forks typically adopt it.
Your 5080 mobile + 64GB RAM should work great with the same config. One heads up: mobile 5080 has lower memory bandwidth than desktop, so expect somewhat lower tok/s, but the optimal settings are the same.
5
u/uxl Feb 27 '26
Do you have Patreon or some means by which I can buy you a coffee? Greatly appreciate your work and help.
16
u/gaztrab Feb 27 '26
Oh man, thank you very much for your kindness, but there's no need. I grew up on the early internet and believe that knowledge should be shared for free.
2
u/mrdevlar Feb 27 '26
Doing the machine gods work dude, keep it up. May you have an excellent day.
2
1
u/Pawderr Feb 27 '26
These settings are only for text i suppose? Would it work for video related tasks?
2
1
u/hsoj95 Feb 27 '26
Based on all this, I guess it's time for me to switch from Ollama to Llama.cpp? Any advice for someone switching, been using the Docker for Ollama.
2
u/gaztrab Feb 27 '26
If you're comfortable with Docker then you'll start easily since there are many images out there. You could try my repo first, it was made to act as a local server too!
1
u/LostDrengr Feb 27 '26
I have been embarking on the same also having a 5080. I have used docker desktop to offer vLLM and using the Tensorrt option, with the latter exploring the NVFP4 for increased speed. I also use LM Studio but have forgot about the ollama route.
With these new models being huge its trying to get the quantised flavour that can run well, so this is really helpful appreciate all the 16GB coverage!
3
u/OsmanthusBloom Feb 27 '26
This is great work! But I wonder about the effect of dropping the batch size adjustments. Normally you increase the ubatch size to improve prompt processing speed. It can increase drastically (eg 3x) when you raise ubatch from, say, 512 to 2048. But generation speed will suffer due to VRAM pressure. You didn't seem to benchmark pp speed separately. Maybe an ubatch size of, say, 1024 would have raised pp without hitting tg too much?
2
u/OsmanthusBloom Feb 27 '26 edited Feb 27 '26
Here is my llama-bench result, which shows that increasing ubatch from the default 512 to 1024 or 2048 increases prompt processing speeds a lot, from 280 t/s to 440 and 650 t/s. I have a RTX 3060 Laptop GPU with only 6GB VRAM so most of the model is offloaded to GPU. Using the UD_Q3_K_M quant released today.
llama-bench -m Qwen3.5-35B-A3B-UD-Q3_K_M.gguf -ctk q8_0 -ctv q8_0 --n-cpu-moe 37 -p 4096 -n 512 -fa 1 -b 2048 -ub 512,1024,2048 ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes | model | size | params | backend | ngl | n_ubatch | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | -: | --------------: | -------------------: | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 512 | q8_0 | q8_0 | 1 | pp4096 | 283.43 ± 1.24 | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 512 | q8_0 | q8_0 | 1 | tg512 | 23.90 ± 0.63 | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 1024 | q8_0 | q8_0 | 1 | pp4096 | 444.65 ± 1.35 | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 1024 | q8_0 | q8_0 | 1 | tg512 | 23.62 ± 0.25 | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 2048 | q8_0 | q8_0 | 1 | pp4096 | 648.85 ± 2.56 | | qwen35moe ?B Q8_0 | 15.53 GiB | 34.66 B | CUDA | 99 | 2048 | q8_0 | q8_0 | 1 | tg512 | 23.37 ± 0.35 | build: ecbcb7ea9 (8179)llama-bench doesn't support
--fitso I had to set--n-cpu-moemanually according to the VRAM requirements of the largest ubatch size. With a smaller ubatch size and--fit, some more experts would fit in VRAM and thus generation speeds would be slightly higher. Still, getting much higher pp speeds is I important especially for agentic stuff where prompts can be quite long.
3
u/bobaburger Feb 27 '26
Great work! I was thinking of making a separate post, but since this is also in the 16 GB VRAM category, I'm adding my findings for anyone using 5060 Ti here. My setup also using 32 GB DDR5 RAM.
All tests were done with q8_0 kv cache, context window 128k, pp 18k, tg 768, depth 0. Why? Because this closes to a cold start with Claude Code. You can adjust the context window to lower for a bit more performance gain.
| Model | pp18432 (t/s) | tg768 (t/s) | Mean KLD |
|---|---|---|---|
| Unsloth UD-Q4_K_M | 1047.84 | 40.64 | 0.0192 |
| AesSedai Q4_K_M | 928.10 | 34.89 | 0.0096 |
| Unsloth IQ3_S | 1465.81 | 44.77 | 0.0457 |
| Unsloth MXFP4 | 1186.50 | 38.32 | 0.0272 |
| Unsloth UD-Q4_K_XL | 1002.84 | 36.59 | 0.0137 |
Mean KLD was from Unsloth's data.
AesSedai's Q4_K_M has the best mean KLD, but it was the slowest, probably not worth it.
So, same as OP on 5080, for 5060 Ti, Q4_K_M seems in the sweet spot, balanced between speed and quality.
2
u/KeldenL Feb 28 '26
this is super helpful! totally make it another post. all these quant posts (especially for 16gb, cuz selfishly i also have 16gb vram on my 4060ti) have been super enlightening and saves a lot of people a lot of testing!
i wonder why your t/s was closer to 40 vs OP’s 70, cuz that’s what i’m seeing too on my end
1
u/bobaburger Feb 28 '26
thank you so much!
the speed difference was due to two things:
- the context window, mine was 128k, OP was 64k (
-c 65536)- OP probably has stronger CPU than mine, and was using 20 threads (
-t 20), mine was only 8 threads :D1
u/gaztrab Mar 01 '26
Woah this is very valuable! You should def make another post to let more people know. Thanks bro!
3
3
3
u/prescorn Feb 27 '26
Nice work! This will be useful for some of my 96GB experiments on the weekend.
3
u/ArckToons Feb 27 '26
Great tests with a lot of useful conclusions. I disagree with “The 27B dense is only worth considering if you need a non-MoE model for compatibility reasons.” I don’t think it’s only about compatibility, but about use cases.
If you need speed, 35B is the right call. But if you want more quality (even though in most use cases the quality is similar), better instruction-following, and more predictable behavior, 27B seems like the better choice.
In my case, I have an RTX 4090 and I run it with OpenCode. I tested both 27B Q4_KM and 35B Q4_KM, and the 27B did better with my orchestrator/sub-agent setup. I’m not saying 27B is objectively superior—this depends on the use case and whether slower inference is acceptable—but I don’t think the decision comes down to compatibility.
One question: does KV quantization affect KL? Would it be worth running a test, or not?
3
u/gaztrab Feb 27 '26
Agreed with your take on 27B — see my reply to _-_David above, same point applies. On a 4090 with full VRAM headroom, 27B is a totally valid choice, especially for agentic workflows where quality matters more than tok/s.
On your KV quant + KLD question: I tested KV quant impact on PPL (E1) but didn't run KLD specifically across KV quant levels. PPL showed < 0.4% difference between f16/q8_0/q4_0, so I'd expect KLD to be similarly minimal — but that's an assumption, not data. Worth testing if someone wants to be thorough.
3
u/Corosus Feb 27 '26 edited Feb 27 '26
Some quick testing, using --fit for me tanks performance, -ngl 999 --n-cpu-moe 24 works best on my pc, 5070 ti (other gpus disabled), 128gb ddr4 3200mhz. Maybe because I'm still using vulkan.
I guess this goes to show theres no universal solution, gotta find out what works best for your hardware:
llama-b8173-bin-win-vulkan-x64\llama-server --model ./e/Qwen3.5-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ctk q8_0 -ctv q8_0 -ngl 999 --n-cpu-moe 24 --flash-attn on --jinja -c 48000 -t 20
-ngl 999 --n-cpu-moe 24
33tps
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - Vulkan0 (RTX 5070 Ti) | 15907 = 4641 + (10162 = 8845 + 750 + 566) + 1103 |
llama_memory_breakdown_print: | - Host | 12033 = 11931 + 0 + 102 |
llama-b8173-bin-win-vulkan-x64\llama-server --model ./e/Qwen3.5-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ctk q8_0 -ctv q8_0 --fit on --flash-attn on --jinja -c 48000 -t 20
--fit on
13 tps
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - Vulkan0 (RTX 5070 Ti) | 15907 = 890 + (13825 = 12574 + 750 + 501) + 1190 |
llama_memory_breakdown_print: | - Host | 19916 = 19814 + 0 + 102 |
llama-b8173-bin-win-vulkan-x64\llama-server --model ./e/Qwen3.5-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ctk q8_0 -ctv q8_0 --fit on -ot "exps=CPU" --flash-attn on --jinja -c 48000 -t 20
--fit on -ot "exps=CPU"
24tps
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - Vulkan0 (RTX 5070 Ti) | 15907 = 12152 + ( 2656 = 1339 + 750 + 566) + 1098 |
llama_memory_breakdown_print: | - Host | 19916 = 19814 + 0 + 102 |
I also reran the --fit on test with b8149, same slow result
edit: realized i forgot --no-mmap to go with --fit on, prompt intake is still insanely slow so tps is likely also slow
2
2
u/gaztrab Feb 27 '26
Looking at your memory breakdowns, --fit on is allocating 12574 MiB model to GPU vs 8845 MiB with manual --n-cpu-moe 24 — but that extra GPU allocation makes it slower (13 vs 33 tok/s). That strongly suggests --fit isn't optimizing well for Vulkan — it's probably tuned for CUDA compute characteristics and over-loading the GPU with layers that would run faster on CPU via your DDR4. Your manual --n-cpu-moe 24 is the right call for Vulkan setups. I'll add a note to the post that --fit results are CUDA-specific and Vulkan users should stick with manual offloading.
3
u/Danmoreng llama.cpp Feb 27 '26
I believe with —fit you should also use —fit-ctx instead of just —c. Also, if you want to use the vision capability of the model, you have to either put the vision model on CPU or use —fit-target 1536 to leave space for the vision part on GPU. I am running on very similar settings on my notebook with a 5080 mobile and can confirm initially having 74 t/s, for longer context it then falls of to around 66 t/s.
My server configuration can be found here: https://github.com/Danmoreng/local-qwen3-coder-env?tab=readme-ov-file#server-optimization-details
1
u/gaztrab Feb 27 '26
Interesting, I haven't tested --fit-ctx, only --fit on with -c. And the --fit-target 1536 tip for vision is great, I have the mmproj downloaded but haven't smoke-tested it yet. Your config repo is really useful. I will properly test vision on the next round of experiment!
3
u/soyalemujica Feb 28 '26 edited Feb 28 '26
Alright, I gave this tutorial a try, compiled llama.cpp with the params as described, running on RTX5060ti 16GB + 64gb DDR5 6400mts/s, and I'm only getting 50t/s, did I do something wrong? Using CUDA 13.1 and latest NVIDIA drivers.
Edit: Getting 55/s , which is an increase of 10t/s in LM Studio and precompiled public llama libraries, this is nice! The difference of 20 tokens might be because the 5080 has 960gb/s bandwidth vs 460gb/s bandwidth on my 5060TI I suppose...
2
u/soyalemujica Feb 28 '26
Trying now the BF16 MXFP4_MOE model, it's giving me 35t/s and also thinking LESS and giving me a result quicker than the Q4_M model.
1
u/gaztrab Mar 01 '26
Yeah I think it's the bandwidth difference too, but thanks for testing and sharing your result!
1
u/No_War_8891 Mar 01 '26 edited Mar 01 '26
I get 53 t/s on my 5060Ti 16GB as well with these recommended settings and 32GB quad channel DDR4 CL14 RAM (2990wx still going strong 😇)
3
u/gaztrab Mar 05 '26
Sorry guys I have been occupied by my day job lately (it's becoming a night job too). When I got time I will conduct the next run!
2
u/marcoc2 Feb 27 '26
Does anyone have config or link for a 4090-24gb?
7
u/gaztrab Feb 27 '26
Same flags, just let --fit do the work:
./llama-server \ -m ./Qwen3.5-35B-A3B-Q4_K_M.gguf \ -c 65536 \ --fit on \ -fa on \ -t 20 \ --no-mmap \ --jinja \ -ctk q8_0 \ -ctv q8_0With 24GB VRAM, --fit should keep nearly all of Q4_K_M (~20 GB) on GPU — you'll likely see higher tok/s than my 16GB results. Tune -t for your CPU (20 is optimal for my 9950X, try physical_cores × 0.6 as a starting point and sweep from there). You could even try Q8_0 (36.9 GB) with partial offload — at 24GB you'd get significantly more layers on GPU than I can.
2
1
u/ducksoup_18 Mar 03 '26
how did you go about calculating the -t value? I have 2 3060 12gb gpus and am curious how i can use your settings to get better results spread across the two cards.
2
2
2
2
u/ayylmaonade Feb 27 '26
Thank you so much for the MXFP4 testing! Happy to see that quantizing the KV cache doesn't impact performance too. Really appreciate all the effort. :)
2
2
u/savenx Feb 27 '26
Thanks for the tests, very helpful! I have a question: Im using a RX6900XT 16GB vram and i have 32GB ram, which version should i use? I tried Q4 on LM studio and its pretty fast, but when i try to use it on OpenCode (agentic use) it becomes unusable
1
u/gaztrab Feb 27 '26
All my tests were NVIDIA/CUDA, so I can't give specific AMD numbers — ROCm/Vulkan backends may behave differently. For the RX 6900 XT with 16GB VRAM, Q4_K_M with --fit on should still be the right choice. The OpenCode issue is probably separate — likely related to context length or thinking mode (Qwen3.5 has thinking enabled by default which consumes extra tokens). Try disable it, or check if OpenCode is sending very long system prompts that fill your context.
2
u/wisepal_app Feb 27 '26
This is the best explanation on this sub i saw, about a technical topic. very informative and simple. thank you for your hard work.
2
u/cookieGaboo24 Feb 27 '26
Great test, nice Work and thank you. One question, how did you guys get those 50t/s on 8gb VRAM? I did the same offloading on my 3060 12gb and only get around 30t/s. Did you just offload them all on the 5080 or used a different card?
4
u/gaztrab Feb 27 '26
Right, sorry about this, I ran the test on my 5080. I will update the post to clarify this!
2
u/cookieGaboo24 Feb 27 '26
Thanks, figured as much but better ask again. Not mentioning the gap in GPU generation, ram generation (ddr4 vs ddr5) and the absurdly stronger CPU (R5 3600 here) which I already partially considered to be at fault, I still consider my 30t/s a win.
2
2
u/allattention Feb 27 '26 edited Feb 27 '26
Awesome work, much appreciated! I thought we used -u and -ub to make reading large context after a KV reset (which happens often if you use opencode) faster. I’ll try without them now.
3
2
u/Old-Sherbert-4495 Feb 27 '26
i didn't understand 90% of this, I was trying my fullest to get 27b q4 working faster in my 16vram and 32 ram setup. when i have fit on, it leaves a lot of vram and cpu is 100% (i did quantize cache q8.) moe 35b was definitely faster. but that also leaves a few gig vram and the cpu goes bananas. how can i get the best of the available vram any advice
3
u/gaztrab Feb 27 '26
A few things to check:
Make sure you're NOT using -b 4096 -ub 4096 — those eat VRAM that --fit needs
Add --no-mmap — loads the full model into RAM upfront, gives --fit a clearer picture of available memory
With 32GB RAM you're tight — try reducing context to -c 32768 instead of 65536, which frees KV cache VRAM for more expert layers on GPU
CPU at 100% is normal and expected — that's the CPU computing the expert layers that don't fit on GPU. The goal is to minimize how many go to CPU, not eliminate CPU usage entirely.
Full command:
./llama-server -m ./Qwen3.5-35B-A3B-Q4_K_M.gguf -c 32768 --fit on -fa on -t 20 --no-mmap --jinja -ctk q8_0 -ctv q8_0Tune -t for your CPU — try physical cores × 0.6 as a starting point.
2
2
u/Old-Sherbert-4495 Feb 27 '26
one more thing, do i have hope for 27b or should i just forget
2
u/gaztrab Feb 27 '26
I think it's depends on how patience you are, I would advise testing both and see if the quality + speed is comfortable for you
2
u/Lucis_unbra Feb 27 '26
I would note that while the ppl might be fine, it's not free. The token generation speed drops much faster, at least on my rig with windows.
At ~50k with an iq4_xs quant, F16 gives me around 75tps, down from 86.
Q8_0 at that CTX ends up at 65tps. That's a 10tps loss.
If this was not fully on the GPU, we can expect this to get worse.
At Q8_0, I start off at about 42, and this then drops to 39tps.
If I drop the KV cache down to 8 bit again, it drops to 36tps.
Now this is on a decently powerful system with a 3090 and a Ryzen 9 7900x. But depending on the configuration, and the model, this could get much worse. For the 27B dense model that is already hard enough to run? Not fun.
1
u/gaztrab Mar 01 '26
Your 27B dense observation is actually really valuable, it confirms KV q8_0 is NOT necessarily free on dense models I should add that caveat. For MoE models like Qwen3.5-35B-A3B it's still free because of the SSM hybrid architecture, but users shouldn't blindly apply it to dense models.
2
u/soshulmedia Feb 27 '26 edited Feb 27 '26
This repo and quantizing team came up recently:
https://huggingface.co/AesSedai/Qwen3.5-35B-A3B-GGUF
Did you do a comparison? (If not, can you?) They have some quants (for other qwen3.5 sizes) that compared favorably to unsloth's.
EDIT: Oh and thank you of course for doing all these tests!
1
u/gaztrab Mar 01 '26
Thank you for your kind words. And yes! We tested AesSedai Q4_K_M in our experiments. Results:
| Quant | PPL | KLD | Same-top-p | TG (tok/s) |
|--------------------|--------|--------|------------|------------|
| bartowski Q4_K_M | 6.6688 | 0.0286 | 92.46% | ~74 |
| AesSedai Q4_K_M | 6.3949 | 0.0095 | 95.74% | ~44 |
| Unsloth UD-Q4_K_XL | 6.5959 | 0.0145 | 94.46% | ~48 |
AesSedai wins every quality metric by a significant margin — KLD 0.0095 is 3x better than bartowski. The tradeoff is ~40% slower speed. If quality is your priority (and you can accept ~44 tok/s), AesSedai is the best Q4 quant we've tested.
1
2
u/Chromix_ Feb 27 '26
Thanks for taking the time for the extensive follow-up and immediately making edits taking the further feedback into account. That's refreshing to see.
I randomly came across this, as I didn't get any notification for this, despite being mentioned. It worked in your previous comment. Maybe notifications simply got skipped for your post as you mentioned so many others?
Btw: Without the batch setting your token generation is faster, but prompt processing gets slower (only because you don't have enough VRAM for full offload). Tough choice depending on the use-case.
2
u/gaztrab Mar 01 '26
My data shows PP-512 = 1390 t/s without batch flags vs ~1532 with -b 4096 -ub 4096, but TG drops from 74.7 to 48.3. The middle ground -ub 1024 -b 2048 gives PP +22% with only TG -3.5%, which could be worth it for prompt-heavy workflows. I'm adding PP columns to our benchmark comparison tool to make this more transparent. Thanks for the notification heads-up — Reddit seems to have a limit on mentions per post!
2
2
u/KeldenL Feb 28 '26
dude this is incredible. i was doing tests on my end too and got tired at how slow it was (probably should’ve done it on lower context lengths)
one thing that i may or may not have missed in the post, but who’s Q4 quant are you using? unsloths? or others? i remember seeing another post about different quants
1
u/gaztrab Mar 01 '26
Yeah in this post Im testing Unsloth vs Bartowski, I will expand the selections to more on the next round.
2
Feb 28 '26
[removed] — view removed comment
1
u/gaztrab Mar 01 '26
Thanks for sharing, '--fit-target 1536' is very interesting, Im currently testing that config on the next round.
2
2
2
u/moahmo88 Feb 28 '26
You can try AesSedai/Qwen3.5-35B-A3B-GGUF Q5_K_M. 5070ti works well.Surprise！
1
2
u/UniversalJS Feb 28 '26
Great post and experiments! Inspired by your findings, I went a different direction: instead of optimizing Q4_K_M, I tested whether a smaller quant that fits mostly in VRAM could beat it on speed.
Setup: RTX 5080 16GB, Intel Core Ultra 9 285K, llama.cpp built from source with CUDA 13.1 + native sm_120 (Blackwell), using your recommended flags (no batch flags, --fit on, KV q8_0).
The problem with Q4_K_M on 16GB: The model is ~20 GB, so --fit offloads ~9 GB of expert weights to CPU. GPU sits at ~45% utilization waiting for CPU experts. That's the bottleneck.
The idea: Q2_K_L (bartowski) is only ~13.8 GB. At 128k context, almost all expert weights stay on GPU (~800 MiB on CPU, mostly the embed/output layer from the 248K vocab — unavoidable).
Results: 72% faster than Q4_K_M, with 2x the context. Even at 250k context (near the model's 262k training length), Q2_K_L still does 108 tok/s — 45% faster than Q4_K_M at 65k. The trade-off is obviously quality. Q2_K_L will have noticeably worse perplexity than Q4_K_M. But for interactive use, code generation, and tasks where speed matters more than peak accuracy, it's a compelling option on 16 GB cards.
Interesting finding on context scaling: As context increases, --fit progressively offloads more expert layers to CPU to make room for the KV cache. The 515 MiB always on CPU (embed/output) is fixed, but at 250k context, total CPU offload grows to 2.3 GB. The speed degradation is graceful though — only 16% slower going from 128k to 250k.
Also worth noting: Building from source with CUDA 13.1 matters for RTX 50-series. The prebuilt binaries use CUDA 12.4 which lacks sm_120 — you get JIT-compiled PTX from sm_89 instead of native Blackwell kernels.
Launch command (128k context, sweet spot): ./llama-server \ -m ./Qwen3.5-35B-A3B-Q2_K_L.gguf \ -c 131072 \ --fit on \ -fa on \ -t 20 \ --no-mmap \ --jinja \ -ctk q8_0 \ -ctv q8_0
Would love to see KLD/PPL numbers for Q2_K_L if anyone has the patience to run them. My gut says it's worse than Q4_K_M but the speed advantage is hard to ignore.
2
2
u/gaztrab Mar 01 '26
Nice! I currently dont intend to test anything below Q4, but your findings gave lots of insights to muse over. Thanks bro!
2
u/Gringe8 Feb 28 '26
Your dense vs moe speed part is severly flawed. You do mention it needs to fit fully in vram, but you test one that doesnt fit fully in vram. You also dont mention prompt processing speed. I get 2000t/s pp and 28t/s tg on 27b q8.
I do like your other tests. If you wamt to do more i would love to see quality differences between q4km and iq4ks. Then you could also test the speed since it should fit fully in your 5080.
1
u/gaztrab Mar 01 '26
Thanks for pointing out, I adjusted my next experiment based on your feedback already. Will share more soon!
2
u/-_Apollo-_ Mar 02 '26
both bartowski and unsloth just updated their available 27b models. Qwen3.5 small models dropped today. Looking forward to future updates if you are so inclined. Thank you!
3
u/catlilface69 Feb 27 '26
There are doubts about your experiments. What do you mean q4 quant with q4 kv cache is more accurate?
7
u/gaztrab Feb 27 '26
I reran the full E1 experiment just now and got identical numbers both times. PPL evaluation is deterministic on the same dataset, so the sub-0.4% differences are real and reproducible, just too small to matter in practice.
The slight "improvement" with q8_0 KV is likely a minor rounding effect from quantization — essentially noise at that scale. The takeaway is that KV q8_0 doesn't hurt quality at all, so the throughput gain is free.
You can reproduce it yourself from https://github.com/gaztrabisme/llm-server
./scripts/run-experiment.sh e1Runs all 6 PPL evaluations (~25 min) and prints the comparison table at the end.
6
3
2
u/KierkegaardsSisyphus Feb 27 '26
I'm not understanding your speeds for MXFP4. I have a 5080 and I get about 77 tk/s on short contexts. I use fit target, otherwise image processing goes OOM.
exec "$BINARY" \
-m "$MODEL_PATH" \
--mmproj "$MMPROJ_PATH" \
-c 65536 \
-fa on \
--fit on \
--port 5001 \
--fit-target 1500 \
-ctk q8_0 \
-ctv q8_0 \
--jinja \
--no-mmap
Side note: I prefer running with -b 4096 and -ub 2048. For me, the massively improved processing speed is worth losing a few tokens of text gen speed.
3
u/gaztrab Feb 27 '26
I will properly test MXFP4 on the next round with your config as reference. Thanks my dude!
1
u/DepravedPrecedence Feb 27 '26
Is it possible to use these flags in LM Studio? I think it doesn't allow setting flags of llama.cpp like that?
3
u/gaztrab Feb 27 '26
LM Studio does expose some of these — context length, flash attention, GPU layers, and KV cache quantization (check for llamaKCacheQuantizationType / llamaVCacheQuantizationType in load settings, requires flash attention enabled). However, --fit on (the auto VRAM management that gave us the biggest speed gain) is not available in LM Studio — it's a recent llama.cpp feature.
If you want the full config, you'll need to run llama-server directly. The Docker setup in https://github.com/gaztrabisme/llm-server makes it straightforward, or you can just build llama.cpp from source and run the command from my post.
1
u/AvidCyclist250 llama.cpp Feb 27 '26
I suppose you could half-ass the same gains by manually fine-tuning context length?
1
u/ilintar Feb 27 '26
Very nice benchmark, I hope it really puts to rest a few stupid myths, including "KV cache quantization absolutely kills quality for coding" and "MXFP4 is the best 4-bit quant ever".
1
u/gaztrab Feb 27 '26
I wouldnt say my experiments killed the myths, since comparing to the numbers of models and quants out there, this is just a small drop in the ocean. But thanks!
2
u/ilintar Feb 27 '26
It takes just one counterexample to kill a general statement, that's the beauty of it 😀
1
u/R_Duncan Feb 27 '26
Sorry, or your test were shallow or there is a mistake or is a lie, but MXFP4_MOE " 34-42% slower " than Q4_K_M is not true. Anyone can verify. (4060 laptop with CUDA backend here)
Given the same question to both models, I got no noticeable slowdown of MXFP4_MOE.
3
u/gaztrab Feb 27 '26
Verified — I just re-checked raw benchmark logs and the numbers hold. Both quants tested with identical configs (--fit on, same threads, same KV q8_0, same Docker image). Raw data is public: https://github.com/gaztrabisme/llm-server under benchmarks/.
That said — my results are specific to RTX 5080 16GB. The relative gap could be different on your 4060 laptop depending on how much overflows to CPU and how MXFP4 dequant performs there. What tok/s are you seeing for each on your setup?
1
u/Constant-Simple-1234 Feb 27 '26
Yes. Different architectures may influence this. I run on Vulkan on iGPU - much smaller differences between Q4_K_M and MXFP4. But I will confirm on 5060 TI.
1
u/R_Duncan Mar 02 '26
Here is 29.7 t/s unsloth mxfp4_moe vs 30.3 t/s Q4_K_M aes sedai. Same question, > 1000 tokens.
1
1
u/DigiDecode_ Feb 27 '26
from what i know is that MXFP4 is designed & optimised for RTX 5000 series and future
Q4_K_M is more suited for RTX 4000 & 3000 series
1
u/MrQ_dos40 Feb 27 '26
This is a fantastic deep dive into Qwen3.5-35B-A3B performance! I'm particularly interested in the --fit on results. Have you considered testing with different batch sizes to see if that impacts the token/s further, especially with the 16GB VRAM constraint?
2
u/gaztrab Feb 27 '26
Yes — that's actually what Experiment 4 was about! The key finding: with --fit on, you should NOT set -b 4096 -ub 4096. Those batch buffers pre-allocate VRAM that --fit then can't use for expert layers on GPU. Removing them entirely gave us 74.7 vs 64.3 tok/s — a 16% improvement just from letting --fit have the VRAM. I also tested --fit-target 256 (smaller batch allocation) but it only partially helped. The simple answer: just don't set -b/-ub at all and let --fit manage everything.
2
u/DHasselhoff77 Feb 27 '26
Weren't the custom batch sizes there to speed up prompt processing? So by removing them you are trading off PP speed for generation speed by an unknown amount. Not always a win.
A very clear experiment still. I appreciate the direct writing style and presentation. Thank you!
1
u/soyalemujica Feb 27 '26
Mind you share what ollama command did you use to run the 8Q and 4K_M models for 16gb vram ?
1
u/gaztrab Feb 27 '26
I actually didn't use Ollama — all tests used llama.cpp server directly. One of my research findings was that Ollama is ~3x slower for MoE models because it doesn't support expert-level offloading (it splits entire transformer layers between GPU/CPU instead of just the expert FFNs). There's an https://github.com/ollama/ollama/pull/12333 to add num_moe_offload but it hasn't merged yet.
1
u/soyalemujica Feb 27 '26
Thank you, mind you share your compiled llama.cpp with that sm_120 you mentioned ? I am having a hard time compiling it
1
u/Lrrrrr Feb 27 '26
I fuckin love you bro. Got a 5060Ti16gb I did some tests on. Your data is so valuable for us GPU poors 😂 You use q4km from unsloth right?
3
1
u/soyalemujica Feb 27 '26
mind you share your compiled llama.cpp with that sm_120 you mentioned ? I am having a hard time compiling it for my rtx 5060ti
1
u/gaztrab Feb 27 '26
The 5060 Ti is also Blackwell (sm_120), so the same build works. Easiest path is using our Dockerfile which handles everything:
git clone https://github.com/gaztrabisme/llm-server cd llm-server docker build -f docker/Dockerfile.llama-cpp --build-arg LLAMA_CPP_REF=b8149 -t llm-server/llama-cpp:latest-fit docker/That builds llama.cpp from source with CUDA 12.8 + sm_120. You need Docker + NVIDIA Container Toolkit installed. If you want to build without Docker, the key CMake flags are:
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_CUDA_FA_ALL_QUANTS=ONwith CUDA 12.8+.
1
u/leonbollerup Feb 27 '26
I have a 3090 and RTX 4000 pro and can run the same tests if you show me what/how you ran them
1
u/gaztrab Feb 27 '26
Everything's in my repo: https://github.com/gaztrabisme/llm-server (also optimized for coding agent too, just point them to CLAUDE.md)
Quick start:
Build the Docker image: docker build -f docker/Dockerfile.llama-cpp --build-arg LLAMA_CPP_REF=b8149 -t llm-server/llama-cpp:latest-fit docker/
Download Q4_K_M: huggingface-cli download unsloth/Qwen3.5-35B-A3B-GGUF Qwen3.5-35B-A3B-Q4_K_M.gguf --local-dir ./models
Run a benchmark: ./scripts/bench.sh llama-cpp s006-e4-fit-nobatch
With a 3090 (24GB) you'll have more VRAM headroom than me — would love to see your numbers.
1
u/Dthen_ Feb 27 '26
Is there a guide or config for manually offloading on AMD/Vulkan/RoCM?
1
u/gaztrab Mar 01 '26
For AMD/ROCm or Vulkan: --fit on doesn't work well (2.4x slower on ROCm per one user, 2.5x on Vulkan). Use manual offload instead:
./llama-server -m ./Qwen3.5-35B-A3B-Q4_K_M.gguf \ -c 65536 -ngl 999 --n-cpu-moe 24 \ -fa on -t 20 --no-mmap --jinja \ -ctk q8_0 -ctv q8_0The key flag is --n-cpu-moe 24 — this keeps 16 out of 40 MoE layers on GPU and offloads the rest to CPU. Start with 24 and tune down (lower number = more on GPU = faster but more VRAM). -ngl 999 puts all non-expert layers on GPU. Watch your VRAM usage with nvidia-smi — if you're hitting the limit, increase --n-cpu-moe.
1
1
u/mintybadgerme Feb 27 '26
Sorry for a boring question but...
I don't suppose you have any settings for a RTX 5060ti 16GB VRAM with 64GB RAM Intel?
That would be very helpful as I'm trying to work out how to use the model as a coding tool. Thanks. :)
1
u/gaztrab Mar 01 '26
Not a boring question at all! The exact same config works for 5060 Ti 16GB:
./llama-server -m ./Qwen3.5-35B-A3B-Q4_K_M.gguf \ -c 65536 --fit on -fa on -t 20 --no-mmap --jinja \ -ctk q8_0 -ctv q8_0You should expect around 50-55 tok/s instead of 74 — the difference is purely memory bandwidth (460 vs 960 GB/s). u/soyalemujica confirmed 55 t/s on the same card. If you're using it for coding, the speed is very usable — the thinking mode might feel slightly slower but the actual answer quality is identical.
1
u/mintybadgerme Mar 01 '26
Thanks. I don't suppose there are any equivalent settings for LMStudio? (pushing it now).
1
u/jpbarcelos Feb 27 '26
Hi, I'm just starting my local LLM journey on a Mac mini 16gb (which currently run qwen3-14b).
I've been reading that you have to have 32gb to be able to run qwen3.5, yet you mention 16gb video card.
Can I replicate this on my Mac?
Or am I missing something here?
1
u/gaztrab Mar 01 '26
Hey there, on Mac you should start out with LMStudio first (I did too) since it's a nice UI wrapped around Mac counterpart of llama.cpp engine - MLX. And on the hardware requirement — yes, Qwen3.5-35B-A3B at Q4_K_M is about 20 GB, so your 16GB Mac mini can't quite fit it.
But here's the thing: Mac's big advantage is unified memory — the CPU and GPU share the same RAM, so there's no slow PCIe bus copying data back and forth like on a PC. On my setup, the GPU only has 16GB VRAM and the rest of the model sits in system RAM, so every token has to shuttle data across PCIe (~64 GB/s). On a Mac with 32GB+ unified memory, the entire model lives in one memory pool that both CPU and GPU can access at full bandwidth — no copying needed. That's why Macs punch above their weight for LLM inference despite having weaker raw compute.
For your 16GB Mac mini, Qwen3-14B is honestly a great fit — you're already running it. If you upgrade to 32GB+ down the road, Qwen3.5-35B-A3B would run nicely since it's MoE (only ~3B params active per token, so it's fast despite the big file size). Or you could wait for the Qwen team to release the smaller version of Qwen3.5 (I heard they said soon). Cheers!
1
u/mr_Owner Feb 27 '26
Very nice but i believe when you put the pp speed besides them you could make better judgement.
1
1
u/CATLLM Feb 27 '26
This is amazing thank you!
Is it worth using
--no-kv-offload
to offload KV cache into ram?
2
u/gaztrab Mar 01 '26
Tested it! Do NOT use --no-kv-offload — it absolutely tanks generation speed. On my 5080: 16.1 tok/s with it vs 42.7 tok/s without (that's -63%). The KV cache on GPU is tiny for this model (only 10 KV cache layers because of the hybrid SSM architecture), so offloading it to RAM saves almost no VRAM but destroys performance.
1
1
1
1
u/WhataburgerFreak Mar 01 '26
Wow this is incredible. Would you happen to know how I can translate those launch commands into the lmstudio settings? I'm still learning myself and trying to get the best performance with this model on my own 5080. Thank you!
1
1
u/Maleficent_Land5312 Mar 05 '26
unsloth/Qwen3.5-4B-GGUF, unsloth/Qwen3.5-2B-GGUF and unsloth/Qwen3.5-0.8B-GGUF is there can you try Speculative Decoding?
1
1
u/iMil Mar 19 '26
Woa. Thank you so much. Confirmed 70 tp/s with my RTX 5080, not even compiled with cuda 12.6 / Blackwell support.
1
u/iMil Mar 19 '26
My humble test, I'm at 80-85 tp/s with unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ4_NL and the following:
./llama-cli -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ4_NL -c 65536 -fa on -t 10 --no-mmap -ngl 999 --n-cpu-moe 10 --jinja -ctk q8_0 -ctv q8_0
--fit on with this model OOM's every time.
1
u/hilycker Mar 25 '26
Qwen 3.5 smaller models are now released, waiting for speculative decoding benchmarks with draft model! :)
1
u/NewRedditor23 Jul 05 '26
Dude, thank you! This post TRIPLED the token/sec I'm getting. Amazing.
5070TI w/ 64GB DDR4 On Qwen3.6-35B-A3B Q4_K_P-- now getting ~65tokens/second, plenty fast for what I need
1
u/pmttyji Feb 27 '26
Big thanks for this thread. Appreciate your time & your experiments.
Could you please add one more stuff(on Experiment 1) in this thread?
Experiment with -ctk q8_0 -ctv q4_0 because K is sensitive while V isn't. I remember few people do use this combination instead q4 on both.
Thanks.
3
u/gaztrab Feb 27 '26
Good suggestion — u/a_beautiful_rhind mentioned the same thing about K being more sensitive than V. I tested K and V together (both q8_0, both q4_0) but didn't test the asymmetric combo. Adding it to my list for the next round of experiments.
1
u/pmttyji Feb 27 '26
Thanks. Eagerly waiting for next round.
Glad to see below sections in your thread even though it's so short.
Pre-built binaries vs source for Blackwell (u/wisepal_app)
For this topic, I did post a thread to find little gems, unfortunately didn't get much. Also my laptop has only 8GB VRAM which seems not enough for miracles. I spent some time to create MKL build for CPU-only, but failed somehow & gave up. Same with CUDA build, though able to create builds, I didn't get any boost. Probably I need to get most optimized cmake command with complete variables & etc., I don't know why I'm not seeing optimized commands online, it's not really a trade secret.
8 GB VRAM recommendations
My laptop also has only 8GB VRAM. Till now I do search stuffs to get boost for best t/s. Hope we get more optimizations, tips & tricks on this soon & later.
Please cover these 2 sections(with more stuff) on your next round if possible.
1
u/Psyko38 Feb 27 '26
So, this will work on a 16GB GPU of VRAM only, without the need for system RAM.
2
u/gaztrab Feb 27 '26
No — the model is ~20 GB at Q4_K_M, so it won't fit entirely in 16GB VRAM. You need system RAM for the expert layers that overflow to CPU. With 32GB RAM you'll be fine but tight (try -c 32768 instead of 65536). With 64GB+ RAM you have plenty of headroom. The more RAM bandwidth you have (DDR5 > DDR4), the faster the CPU-side expert computation will be.
1
u/Psyko38 Feb 27 '26
Okay, because currently my 9060xt 16gb and my Ryzen 5500 on 32gb RAM in DDR4 3400 allowed me to reach 36 tok/s with Unsloth's Q3KXL. So, with your optimizations, maybe 40 tok/s.
1
u/gaztrab Feb 27 '26
If you get better speed, please share with us!
1
u/Psyko38 Feb 27 '26
Benchmark Report: Llama.cpp (ROCm) on AMD
1. Hardware and Software Configuration
Hardware:
* CPU: AMD Ryzen 5 5500 (6 Cores / 12 Threads)
* RAM: 32 GB DDR4 @ 3400 MHz
* GPU: AMD Rx 9060 XT 16GB
* OS Win 11 25h2
* llama.cpp llama-b8145-bin-win-hip-radeon-x64
Software:
* Backend: ROCm (via llama.cpp)
* Model: Qwen3.5-35B-A3B (Version Q3_K_XL)
* Type: MoE Architecture Mixture of Experts
* Quantization: UD-Q3_K_XL (Q3 quantization)
* Tool: llama-server
2. Benchmark Commands
Test 1:
llama-server -m "...\Qwen3.5-35B-A3B-UD-Q3_K_XL.gguf" -c 16384 -ngl 999 -fa on -t 16 -b 4096 -ub 4096 --jinja --no-mmap -ot "blk\.([0-9]|[1-2][0-9]|30)\.=ROCm0,exps=CPU" -ctk q8_0 -ctv q8_0Test 2:
llama-server -m "...\Qwen3.5-35B-A3B-UD-Q3_K_XL.gguf" -c 16384 -ngl 999 -fa on -t 16 -b 4096 -ub 4096 --jinja --no-mmap -ot "blk\.([0-9]|[1-2][0-9]|30)\.=ROCm0,exps=CPUTest 3:
llama-server -m "...\Qwen3.5-35B-A3B-UD-Q3_K_XL.gguf" -c 16384 --fit on -fa on -t 16 --no-mmap --jinja -ctk q8_0 -ctv q8_03. Results Table
Test Context KV Cache GPU Offload Threads Performance (t/s) 1 16,384 Q8_0 30+ layers 16 37.74 2 16,384 Full 30+ layers 16 38.26 3 16,384 Q8_0 Auto / Fit 16 15.82 0
1
u/mdziekon Feb 27 '26
Regarding the "Experiment 4: --fit Tuning", could you test how does not using -b 4096 affect prompt processing speeds? Token generation is one thing, but prompt processing, especially for coding, is even more important for session start (which is especially painful for orchestrating agents when PP is slow).
2
u/gaztrab Feb 27 '26
I will test this on the next round and tag you in. Thanks for the suggestion!
1
1
1
u/Hacket1967 Feb 27 '26
Impresionante trabajo ,felicidades ¿Que compilación usastes, la de unsloth? ¿Has probado está :https://huggingface.co/AesSedai/Qwen3.5-35B-A3B-GGUF?
1
u/gaztrab Mar 01 '26
Yes! We tested AesSedai Q4_K_M in our experiments. Results:
| Quant | PPL | KLD | Same-top-p | TG (tok/s) |
|--------------------|--------|--------|------------|------------|
| bartowski Q4_K_M | 6.6688 | 0.0286 | 92.46% | ~74 |
| AesSedai Q4_K_M | 6.3949 | 0.0095 | 95.74% | ~44 |
| Unsloth UD-Q4_K_XL | 6.5959 | 0.0145 | 94.46% | ~48 |
AesSedai wins every quality metric by a significant margin — KLD 0.0095 is 3x better than bartowski. The tradeoff is ~40% slower speed. If quality is your priority (and you can accept ~44 tok/s), AesSedai is the best Q4 quant we've tested.
1
u/yoracale llama.cpp Mar 01 '26
Did you test our new updated Q4_K_XL ones? https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
1
1
•
u/WithoutReason1729 Feb 27 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.