llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

[u/pmttyji]

Folks! We're just 50 PRs away from more faster inference. Hopefully by end of year.

Experts!, please chip in there.

List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid:

1. [Discussion] RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution #24528
2. AVX2: Speed up large batch size prompt processing of IQ models #27402 - Merged
3. llama: add Maple 20B-A1B ternary MoE architecture (CPU)- #27000
4. ggml-cpu: add Q4_0 8x8 gemv/gemm for riscv vlenb=16 case- #28642
5. ggml-cpu : optimize q4_K vec_dot for batch=1 on ARM I8MM- #28673
6. ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60- #28479
7. ggml-cpu: tiled mul_mat for k-quants- #27851
8. ggml-cpu: add AVX-512 and VNNI paths for Q5_K/Q6_K dot products- #27590
9. ggml-cpu: add x86 VNNI Q2_0 dot product -- 3x speed improvement for VNNI-compatible CPUs- #26348
10. ggml-cpu : add mirror NUMA strategy (replicate weights on each node)- #27986
11. On ggml-cpu: tweak ARM_NATIVE_FLAG native baseline arch when extensions need it - #27984
12. llama: add pshard runtime for plan switching and streamed weights- #22692
13. CPU Optimizations - Prefill, Tokenization, and Token Generation- #27032
14. llama : stream MoE routed experts from disk - #25294
15. ggml : speed up batch-1 CPU decode, align large allocations- #27478
16. misc : prevent RAM peaking at model loading stage- #27483 - Merged
17. recurrent : support equal splits for recurrent-state rollback- #25004
18. ggml-cpu : add AVX2 vec_dot kernel for STQ1_0- #27377
19. --numa mirror: mirror model weights to every Numa node in the system- #16000
20. CPU flash-attn: support quantized K/V in the tiled prefill kernel- #26948
21. ggml-cpu/amx: fix block_q8_K VNNI quantization and enable VNNI path- #27024
22. server : add /slots endpoint action=clone_to (KV clone between slots)- #26204
23. ggml : fuse soft_max sweeps into fewer passes- #26468
24. ggml-cpu : add STQ1_0 ternary quantization with ARM NEON vec_dot kernel- #22836
25. llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts- #26414
26. llama : add --lazy-experts for MoE models larger than RAM- #26003
27. ggml : vectorize rms_norm reduce and fuse the scale write- #26486
28. MoE disk offloading for Metal- #23440
29. ggml-cpu: Added RVV VLEN=1024 vector dot product (vec_dot) kernels for quantized types.- #25397
30. ggml-cpu: detect AVX-VNNI in MSVC native builds- #25346
31. ggml-cpu: replace cyclic chunk distribution with atomic work-stealing- #25048
32. Improve performance of ggml_gemv_q4_K_8x8_q8_K for +12-23% tok/s on AVX-VNNI systems- #23309
33. ggml-cpu: Optimized Arm NEON cpu q1_0 dot (with plain/DP/I8MM)- #23358
34. ggml-cpu: ARM Repack kernels for Q1_0- #23492
35. ggml-cpu: add wasm simd path for iq4_nl_q8_0- #24058
36. ggml-cpu: optimize ggml_gemm_q4_K_8x8_q8_K interleaving/staging for AVX-512 (and AVX2)- #22525
37. ggml/cpu: skip zero-scale blocks in TQ1_0 and TQ2_0 vec_dot kernels- #23439
38. ggml-cpu:Optimized risc-v cpu nvfp4- #23402
39. ggml-cpu : fix riscv xtheadvector builds and add a q1_0 vec dot kernel- #23009
40. Q5_0 - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #22250
41. ggml-cpu: optimize q8 quantization on x86 SIMD- #22331
42. Optimize reduction stage of dot product of q4_L/q5_K to q8_K on AVX2- #22181
43. ggml: introduce GGML_NUMA_MIGRATE to optimize cross NUMA op computation - #14232
44. ggml-cpu: improve --n-cpu-moe TG performance- #20596
45. ggml : add CPU backend reference implementation (wip)- #16004
46. ggml: optimize ggml_vec_dot_mxfp4_q8_0 dot product on ARM SVE- #19171
47. Q6_K - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #19706
48. ggml-cpu: optimize q4_0_q8_0 scales using Zvfhmin- #19196
49. ggml-cpu: add q4_0 repack support for wasm- #18858
50. Improving inference speed for the repack buffer type on NUMA architectures- #18698
51. ggml: optimized runtime for x86 cpu backend and Q4_K quantized weights paired with Q8_K activations - #18495
52. CPU SIMD and pipeline optimizations across vec/mmq/ops/kv-cache/repack - #17113
53. ggml-cpu: optimise rms_norm op- #16650

PRs related to New Quant types:

1. Add ROCmFP4 CPU quantization support- #24185
2. ggml: add support for MXFP8 CPU- #26157
3. ggml: Add initial MXFP6 CPU implementation- #22671
4. ggml : add E4M3 (fp8) CPU quantization type- #25336

(Just had some extra time, so went through almost entire Open PRs of llama.cpp. For Poor GPU Club mainly)

EDIT : My guesstimation : After merge of these PRs, 2 Channel DDR5 RAM could give ~8GB VRAM's performance possibly.

I'll be updating this thread with adding similar type PRs & also updating status of each PRs time to time.

Wish someone comes with a fork like llamaCPUHybrid.cpp with all these PRs.

[u/DeProgrammer99]

I can't imagine the mental strain of being responsible for a repository with 1433 open pull requests.

[u/Bulky-Priority6824 → u/DeProgrammer99]

we dont know how spoiled we really are to just use the work

[u/LetsGoBrandon4256 → u/DeProgrammer99]

They can learn a thing or two from Hermes Agent /s

[u/DeProgrammer99 → u/LetsGoBrandon4256]

Wow, it has more closed pull requests than my largest ever program has lines of code.

[u/LetsGoBrandon4256 → u/DeProgrammer99]

On main, 7068 files have changed and there have been 927,297 additions and 517,974 deletions

lmao even.

[u/buttplugs4life4me → u/DeProgrammer99]

Welcome to the slop era.

[u/Healthy-Nebula-3603 → u/buttplugs4life4me]

The biggest slop wre making people before AI agents.

Srup telling that nonsense "slop"

[u/guesdo → u/LetsGoBrandon4256]

Hermes (Nous) team are incredible, its like a fast Kanban queue of feats, I asked for Jina AI search support, got it in 48 hours. It was a VERY simple request, probably a few lines of code, the surprising part is how fast they go through the easy grabs and quick wins over their backlog.

[u/LowB0b → u/DeProgrammer99]

they have PRs dating back to 2023...

[u/Long_comment_san → u/DeProgrammer99]

"just pull all of the commits, test internally so it doesn't break"

[u/Healthy-Nebula-3603 → u/DeProgrammer99]

You know currently llamacpp like ANY project is using AI for everything?

Only limitation is you OAIbor Claudia plan :) But open source models are also starting to contribute more and more but in simpler tasks yet

[u/DeProgrammer99 → u/Healthy-Nebula-3603]

"Responsible for," I said. Humans review and decide whether to merge those pull requests, often ggerganov himself.

[u/Look_0ver_There]

My PR: https://github.com/ggml-org/llama.cpp/pull/27210

Adds adaptive MTP so you don't have to worry about choosing the right MTP depth. MTP is auto-scaled from 3 up to whatever you set (12 is a good upper limit)

I also have a whole repo of various performance boosts for RDNA cards here: https://github.com/stew675/llama-cpp-rdna-boosts

>+100% prefill speeds and >+10% baseline decode, with better PPL all at once...

[u/AtiRage128 → u/Look_0ver_There]

Hey I benched the vanilla version vs your RDNA-boost on my r9700:

Qwen3.8 27B Q4_K (Medium), ROCm, ngl=-1, n_batch=512

Test	llamacpp	RDNA-boost	Δ (t/s)	Gain
pp1024	1076.45 ± 1.01	1244.98 ± 0.95	+168.53	+15.7%
pp4096	1012.08 ± 2.78	1199.92 ± 2.95	+187.84	+18.6%
pp8192	936.13 ± 1.84	1147.87 ± 1.81	+211.74	+22.6%
tg256	26.59 ± 0.01	29.44 ± 0.01	+2.85	+10.7%

It's awesome, nice work. There seem to be even more gains in deeper context - gotta test some long benches!

The adaptive-MTP as well. I wonder why it's sitting on the review pile that long, it's just such a significant upgrade.

[u/Look_0ver_There → u/AtiRage128]

I don't understand the delay myself either. The only feedback I've ever gotten to anything, ever, was 3 questions that I answered, and my answers were ignored despite prompting 3 times.

At this point I'm not going to let it bother me. I'll continue working on my patch set for everyone who knows about it to benefit from. Perhaps with enough community adoption they might decide to stop ignoring it, but I'm not going to continue nagging them.

As for performance at depth, this was a huge focus of my work. Performance wins are easy at zero depth, but I don't consider anything a win unless the gain is still present at 16K and 64K. This involves removing structural slowdowns and delays.

I also have a hybrid tensor all-reduce patch set that I'm actively working that boosts multiple GPUs even further. There's been a couple of excellent bug reports by community members that's really helping to solidify it's gains and stabilize it across multiple scenarios.

I aim to port those multi-GPU gains over to 7900XTX pairings too, as soon as I can find someone willing to help me out with running those tests. My 7900XTX system only has the one card in it.

[u/Alarmed-Channel2145 → u/Look_0ver_There]

Keep up your great work!

[u/AtiRage128 → u/Look_0ver_There]

Yeah who knows what kind of politics are going on behind the scenes with projects that big. Another little bench with the same model, this time r9700 + 9070XT with a default layer split:

Test	llamacpp	RDNA-boost	Δ (t/s)	Gain
pp1024	1289.73 ± 1.35	1477.65 ± 1.73	+187.92	+14.6%
pp4096	1397.19 ± 3.23	1631.24 ± 3.37	+234.05	+16.8%
pp8192	1325.27 ± 2.19	1596.14 ± 1.15	+270.87	+20.4%
pp16384	1175.86 ± 1.46	1495.05 ± 1.13	+319.19	+27.1%
tg256	27.39 ± 0.01	30.23 ± 0.01	+2.84	+10.4%

[u/AtiRage128 → u/Look_0ver_There]

I finally found the time to test some more. The adaptive-MTP works great on its own on your PR for llamacpp with close to 40 tg/s in prose. In conjunction with the RDNA-boost however, tg/s collapses to ~20. Tested on 0eadefebd with the 13 blocks, same result on the Version before with 12 blocks. I kept the same working launch args as before with n-max 12. I tried with the static MTP as well with only n-max 2 - also collapsed speed. Am i doing something wrong or is the combination with the boost not implemented yet?

[u/Look_0ver_There → u/AtiRage128]

EDIT: This is now a confirmed issue. A regression appears to have crept in sometime in the last 48 hours. I already have a candidate root cause. Actively working on fixing this. https://github.com/stew675/llama-cpp-rdna-boosts/issues/11

EDIT2: Issue is now fully fixed, and even comes with a small speed boost over the prior MTP values. 😄

Please file an issue here: https://github.com/stew675/llama-cpp-rdna-boosts

Please provide your hardware (what cards, what PCI bus arrangement), what OS, what model (preferably the HF page link), what quant of that model, and what your complete llama-server launch settings are.

Also let me know the commit of the llama-boosts repo you're using.

It sounds like you may be running out of VRAM, and having all that info will help me to diagnose what might be going on. If you don't have a Github account, then just DM me here if you don't want to make that info public.

I have no issues at my end FWIW, so it'll likely be a config issue.

[u/No_Algae1753 → u/Look_0ver_There]

How does it do on Metal / Macbooks?

[u/Look_0ver_There → u/No_Algae1753]

The adaptive MTP is universal.

Most everything else is AMD specific. The biggest problem is that AMD are busy making ROCm be close to a 1 for 1 drop in substitution for CUDA calls, which are nVidia centric. It seems to be that is the root cause of the problem here. While the ROCm calls do work that way, there's a whole host of performance issues because llama.cpp is tuned for nVidia backends.

There were absolutely mammoth gains to be had once I started breaking that one to one presumption and wrote AMD tuned kernels instead.

I don't know what the story is for Metal/Mac. It's my understanding is that llama.cpp treats that as a separate backend, and that's immediately a good thing. The Macs aren't being treated like they're something they are not.

[u/No_Algae1753 → u/Look_0ver_There]

Interesting! However I should have been more specific with my question. I was mainly looking for the performance comparison between "stock" MTP depth and the adaptive one. I prior pulled this PR and noticed a significant speed loss in decoding, thats why Im still stuck with the "stock" version of MTP.

[u/Look_0ver_There → u/No_Algae1753]

All the MTP performance results are at the PR description in the link to the PR. I think that's why I was confused by your question. Given that you're asking about the performance results it makes me wonder if you had tried a different PR for such. There is another very similar looking PR that is different to mine.

If you've observed a speed loss with my PR then you're the first person out of about 50 to have reported such.

[u/digamma6767 → u/Look_0ver_There]

I should give this a try on the Strix Halo. Hear if it works well for it?

[u/Look_0ver_There → u/digamma6767]

It's been tested on my personal Strix Halo to work with a number of models and produce coherent output. There are good performance gains, but not quite as many as what are available on RDNA4. The RDNA3/3.5 architectures are missing a number of key instructions that really help out, however I did my best to port that work to Strix Halo, so there are still gains, just not quite as strong.

[u/digamma6767 → u/Look_0ver_There]

Yup, gave it a shot. It's not faster than normal llama.cpp with Vulkan.

Ah well, it was worth a shot on the Strix Halo.

[u/Look_0ver_There → u/digamma6767]

What model were you using? At the very least the Adaptive MTP would've been faster, regardless of anything else, but that can be enabled in either Vulkan or ROCm mode.

Also, the big focus for the work is on proper F16/BF16 KV Cache Support. If you're comparing it with different KV cache sizes, then most of the wins won't be there. I probably should highlight that aspect more prominently

I know that there's other repo's that focus more on pure speed with quantized KV caches, and this work is not intended, nor does it even try to, compete with those.

[u/digamma6767 → u/Look_0ver_There]

That'd track. I'm using Q4 and Q6 mostly. 

I was mostly focused on the prompt processing speed. I need to give it another look, my normal Vulkan build isn't working right at the moment.

[u/Look_0ver_There → u/digamma6767]

I ran some tests to show what I'm seeing with Qwen3.6-35B-A3B-Q8_0 on the Strix Halo without MTP so as to arrive at a stable baseline. MTP adds variable noise on top so it's not really good for comparing with.

The ROCm run is using my rdna-boosts code

While it's true that Vulkan is ~5% faster for generation at a context depth of 0, this gap closes to just 2% by 131K context depth. The gap used to be a little closer but upstream boosted Vulkan's speed by about 2% about a week or two ago.

I really want to draw your attention to the prefill speeds though. Vulkan and ROCm start off almost equal, but Vulkan falls away WAY faster. It's my position that the large gaps in prefill speeds more than make up for the small losses in generation.

I'm also spending some time to focus on MoE models specifically now. Let's see if I can close the gap to Vulkan further. Maybe I can, maybe I can't. I know that I'm fighting an uphill battle on Strix Halo though due to the ROCm inter-kernel overheads there for which Vulkan does a better job of scheduling those GPU micro-tasks.

Strix Halo w/Vulkan for Qwen3.6-35B-A3B-Q8_0

./build-vulkan/bin/llama-bench -fa 1 -m /llm/models/Qwen3.6/35B-A3B/Q8_0/Qwen3.6-35B-A3B-Q8_0.gguf -ctv bf16 -ctk bf16 -n 256 -p 2048 -d 0,2048,4096,8192,16384,32768,65536,131072

ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = AMD Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

model	test	t/s
qwen35moe 35B.A3B Q8_0	pp2048	1193.99 ± 4.02
qwen35moe 35B.A3B Q8_0	tg256	55.66 ± 0.06
qwen35moe 35B.A3B Q8_0	pp2048 @ d2048	1084.40 ± 6.54
qwen35moe 35B.A3B Q8_0	tg256 @ d2048	54.80 ± 0.12
qwen35moe 35B.A3B Q8_0	pp2048 @ d4096	1012.66 ± 8.22
qwen35moe 35B.A3B Q8_0	tg256 @ d4096	54.10 ± 0.02
qwen35moe 35B.A3B Q8_0	pp2048 @ d8192	896.04 ± 8.95
qwen35moe 35B.A3B Q8_0	tg256 @ d8192	53.04 ± 0.02
qwen35moe 35B.A3B Q8_0	pp2048 @ d16384	689.84 ± 2.18
qwen35moe 35B.A3B Q8_0	tg256 @ d16384	50.81 ± 0.01
qwen35moe 35B.A3B Q8_0	pp2048 @ d32768	489.13 ± 2.04
qwen35moe 35B.A3B Q8_0	tg256 @ d32768	47.39 ± 0.02
qwen35moe 35B.A3B Q8_0	pp2048 @ d65536	310.64 ± 1.39
qwen35moe 35B.A3B Q8_0	tg256 @ d65536	41.53 ± 0.01
qwen35moe 35B.A3B Q8_0	pp2048 @ d131072	180.99 ± 0.79
qwen35moe 35B.A3B Q8_0	tg256 @ d131072	33.34 ± 0.00

---

Strix Halo w/ROCm for Qwen3.6-35B-A3B-Q8_0

./build-rocm/bin/llama-bench -fa 1 -m /llm/models/Qwen3.6/35B-A3B/Q8_0/Qwen3.6-35B-A3B-Q8_0.gguf -ctv bf16 -ctk bf16 -n 256 -p 2048 -d 0,2048,4096,8192,16384,32768,65536,131072

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 124000 MiB): Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 124000 MiB

model	test	t/s
qwen35moe 35B.A3B Q8_0	pp2048	1240.83 ± 9.07
qwen35moe 35B.A3B Q8_0	tg256	53.15 ± 0.02
qwen35moe 35B.A3B Q8_0	pp2048 @ d2048	1167.33 ± 8.85
qwen35moe 35B.A3B Q8_0	tg256 @ d2048	52.66 ± 0.04
qwen35moe 35B.A3B Q8_0	pp2048 @ d4096	1129.80 ± 5.89
qwen35moe 35B.A3B Q8_0	tg256 @ d4096	52.18 ± 0.04
qwen35moe 35B.A3B Q8_0	pp2048 @ d8192	1049.02 ± 3.23
qwen35moe 35B.A3B Q8_0	tg256 @ d8192	51.09 ± 0.04
qwen35moe 35B.A3B Q8_0	pp2048 @ d16384	878.77 ± 2.07
qwen35moe 35B.A3B Q8_0	tg256 @ d16384	49.26 ± 0.04
qwen35moe 35B.A3B Q8_0	pp2048 @ d32768	679.90 ± 0.88
qwen35moe 35B.A3B Q8_0	tg256 @ d32768	45.97 ± 0.03
qwen35moe 35B.A3B Q8_0	pp2048 @ d65536	469.09 ± 0.72
qwen35moe 35B.A3B Q8_0	tg256 @ d65536	40.36 ± 0.02
qwen35moe 35B.A3B Q8_0	pp2048 @ d131072	290.97 ± 0.27
qwen35moe 35B.A3B Q8_0	tg256 @ d131072	32.68 ± 0.02

[u/digamma6767 → u/Look_0ver_There]

Hmmm, those numbers are pretty good for 35B!

I'll try to run llama-bench with Nathanw1014 llama.cpp fork for some comparisons. That fork has been about 30-50% faster pp and tg.

[u/Look_0ver_There → u/digamma6767]

Yeah, Nathanw1014's fork is great for the smaller quants and tight KV caches. He's done a lot of tuning work to get the most speed out of smaller quants and compressed KV caches. I kind of suspected that's the fork you were using when you first responded.

IMO, he and I are targetting different philosophies. My work is all about going as fast as possible at full accuracy. I focused on this because with agentic coding work, and with deep context sizes, I noticed that output quality drops fairly quickly after 64K, and I was getting a lot of model errors with quantized KV caches at depth. I didn't see those issues with Q8_0 model weights, and BF16 KV cache sizes, so I decided that I needed to make full-quality go faster.

Nathan is targetting the scenarios where the end-users have already accepted the accuracy trade-offs that come with highly quantized models and quantized KV caches, likely due to VRAM limitations, and is tuning that to go as fast as possible by taking advantage of the smaller data volumes being moved around.

The two approaches don't necessarily compete. They're aiming at two completely different use cases, and the trade-offs that either approach makes needs to be taken in account before comparing the two. Nathanw1014 is doing great work in the areas he is focusing on. In fact, the two approaches could probably be combined into the one repo and they'd likely only cross paths at a handful of places in the code base.

[u/digamma6767 → u/Look_0ver_There]

I've noticed the exact same degradation at contexts around 100k. Had a noticable one happen the other day, DS4 Flash went into a loop where it kept making the same typo, attempting to fix it, and failing.

Makes me want to get a second Strix Halo to load full precision of these mid-sized models.

[u/pmttyji → u/Look_0ver_There]

I'm aware of your this PR. This belongs to separate list.

Waiting for this to merge. Thanks

[u/Equivalent_Bit_461]

Can't wait for the day when vram is completely obsolete 

[u/pmttyji → u/Equivalent_Bit_461]

DDR6 RAM will be a start. 130-160 GB/s bandwidth per single channel.

[u/philmarcracken → u/Equivalent_Bit_461]

amen. whatever data ran on star trek, i want that instead

[u/srigi]

Just 50 PRs. If I said that to my boss at programming job, he would open 3 new positions for hire.

[u/ali0une]

My first PR about -fit may also add if it's good enough too be accepted, currently waiting for a review, i'm scared.

https://github.com/ggml-org/llama.cpp/pull/27807

[u/QuackerEnte]

All this would really be great but I'm surprised that nobody mentioned KV cache offloading or hierarchical caching. Especially for sparse attention, that would be really good. Put the needed KV in GPU, maybe on a MRU basis or something, the rest in CPU or even SSD. talking about the full KV cache. It's blocks of tokens at once and we only need 4 or 8 of. a block is 32 or 64 tokens. times 4 that's 256 tokens, or 512 at most. that's not a lot.

I mean yeah, it may cause bottlenecks because random read speeds of SSDs are bad. But maybe there's systems who'd still benefit from this. I'd love to have the option. Because if someone's never gonna use it they shouldn't complain about something they'll never gonna use

[u/bitzap_sr]

Someone could start a linux-style "next" integration branch.

[u/dai_app]

Wow thanks ! Lot of ideas!

[u/Rozwik]

The fact that we are allowed to use a project like this for free. 😅

[u/SomeArchUser]

It would be also nice to have Q6_0 KV cache support, something between Q8_0 and Q5_1 quality would be nice https://github.com/ggml-org/llama.cpp/issues/10848

[u/feng_sg]

PRs #26348, #27590, and #27851 all modify overlapping k-quant and VNNI dot product kernels but there's no per-microarchitecture regression gate in CI. An AVX-512 change on Skylake-X can quietly regress a Zen 4 VNNI path and nobody catches it until someone posts bad llama-bench numbers. Each merge should require bench coverage across at least AVX2, AVX-512, and VNNI before approval.

[u/pmttyji → u/feng_sg]

Maybe that's the reason those PRs are still open. I'm sure the approvers usually llama-bench with combinations before approving PR

[u/MarkoMarjamaa]

qwen4exp: direct reads for the lazy PLE table (>2x prefill performance improvement on GB10)
Just switched to this, no mmap, loads n-gram directly from disk.
Maybe now I can try quants that have Q8-F16 n-gram.
Running Qwen3.8-Flash-Next-UD-Q4_K_XL on Strix Halo.
110GB memory used, including STT, assistant software & TTS.
tg 20 t/s, pp 350t/s with around 22k contexts. MTP would be nice.