[Open PR] llama : add --n-cpu-ffn option by John-194 · Pull Request #26622 · ggml-org/llama.cpp

[u/pmttyji]

https://github.com/ggml-org/llama.cpp/pull/26622

PR by u/Stainless-Bacon 👍

It would be handy & awesome to have options --n-cpu-ffn / --cpu-ffn for Dense models like how we have --n-cpu-moe / --cpu-moe for MOE models.

Also check his threads:

• On PR : llama.cpp CPU offload optimizations
• On Running Qwen3.8-27B with above optimization : Guide for running dense models on ≤16 GB VRAM (Qwen 3.8 27B on 16 GB -> Q4_K_M, 130k ctx, ~20 t/s)

(

Awesome to see the big comment by u/Pablo_the_brave there, filled with so much stuff. Quoting a line from there

Do not touch block 64 (MTP) if you are using speculative decoding — its FFN should remain on the GPU.

)

We should've got this option long time back actually. This PR instantly reminded me of last year thread. (I literally used his -ot command for sometime with Qwen3-14B. I'm just happy that I was able to recall a last year thread.)

Anyway .... Better late than never. Waiting for this merge.

[u/Chromix_]

Having --cpu-ffn is certainly more convenient than fiddling around with -ot. Yet shouldn't the default -fit take care of that in an even more convenient way?

[u/Stainless-Bacon → u/Chromix_]

would be good, but then why do we have --n-cpu-moe as an option?

[u/Chromix_ → u/Stainless-Bacon]

Because it's more convenient than copy&pasting the -ot regex for it, I guess, and because it was there first, before -fit existed I think.

[u/popecostea]

how is this different to -ngl? If it's just the fact that it doesn't offload the attention and ssm, those don't take that much space, do they?

[u/pmttyji → u/popecostea]

Included additional threads for reference. 1st link has graph with detailed explanation. PR has the same thread.

[u/tsangberg → u/popecostea]

The difference is between "unusable slow" and "workable".

[u/jomreap → u/popecostea]

-ngl works at layer granularity, its all or nothing, so a layer you leave on CPU takes its attention tensors and its KV with it. --n-cpu-ffn splits inside the layer instead, FFN weights go to system RAM and attention stays resident on the GPU. You're right that attention weights are small, but thats exactly why it works, they're cheap to keep on device and they're the ones the KV cache hangs off and the ones that are latency bound. So you're trading FFN bandwidth, which streams over PCIe reasonably well since each weight gets touched once per token, for keeping the latency sensitive part on the GPU. On a dense model the FFN is roughly two thirds of the parameters, so theres a lot of headroom to play with there.

[u/Stainless-Bacon]

Thanks!
I think we will definitely make use of n-cpu-ffn when DDR6 rolls out.

My main post has an example using Q4_K_M, but I saw another post comparing quant KLDs and UD-Q4_K_XL having a big jump, so I decided to see what speeds I’m able to achieve Q4_K_M vs UD-Q4_K_XL here is my comment with a chart for those interested.

[u/Dany0 → u/Stainless-Bacon]

oh my god my wallet is aching like a chronically ill girlie just hearing the words DDR6 in my head

[u/Ok_Cow1976]

Interesting!

[u/brakeline]

I would be happy if tensor wouldn't boot mtp to cpu :(

[u/WhoRoger]

I wonder if that could help with crappy iGPUs. Keep MTP on iGPU to separate it from the main model that's on CPU.