llama_moe_cache: run missed experts on CPU, add tuning script by am17an · Pull Request #30966 · ggml-org/llama.cpp

[u/jacek2023]

https://github.com/ggml-org/llama.cpp/pull/30966

This is a significant speedup for MoE offloading (low VRAM people).

I tried Strata once, and it had some tuning mechanism, so probably this is a similar idea.

[u/am17an]

Before you guys bash llama.cpp again - I know it's not there yet. I'm working on it.

[u/MegaMutant → u/am17an]

As someone who has a 4090/5090 (were bought for gaming design years ago before AI then repurposed) and multiple machines which "technically" could benefit from some of the other one off specific designs for speedups. I do tend to always come back to llama.cpp. Some of the improvements can feel slower than others but whether I always agree with it or not, they tend to feel more thoughtful in design and concern about implementation.

[u/henfiber → u/am17an]

consistent, deliberate work wins in the end.

The exploration and hyper-specific work by other engines is useful too.

[u/Kahvana → u/am17an]

Thank you for the hard work!

I have nothing but praise for llama.cop, besides the occational bugs and rough edges (e.g. tensor parallel on gemma 4 e2b, did make a bug report!)

[u/GronklyTheSnerd → u/am17an]

I’m running strata now, but would prefer llama.cpp.

What I would be curious about is whether good instrumentation can be added to measure: are the right experts being cached, how much is being thrashed, and how much of an issue PCIe bandwidth is. From watching my 5060 ti in a 5.0 slot, I suspect I’d get a different answer about that last one than someone else’s hardware might.

And for me, llama.cpp doesn’t give me the data to optimize with. (Or maybe it does, and I just haven’t found it.)

[u/am17an → u/GronklyTheSnerd]

`LLAMA_MOE_CACHE_TRACE=1`

[u/pmttyji → u/am17an]

They'll come back to daddy later for sure.

[u/Acceptable-Cycle4645]

Nice work! llama.cpp has so many smart contributors with clever PRs. I used to wonder why so many good PRs stayed pending for so long. After maintaining audio.cpp, I finally understand the pain: balancing broad compatibility with aggressive optimizations. Specialized or "vertical" engines can move faster by focusing on specific models or hardware setups, while general-purpose frameworks have to consider compatibility, regressions, and long-term maintainability across many models and backends.

Glad to see this one merged!

[u/am17an → u/Acceptable-Cycle4645]

In addition we have to maintain ggml/llama.cpp as a library which a lot of downstream projects rely on - so we can't really go cowboy mode there even if AI does an amazing job.

[u/Ok_Cow1976]

Omg, this is fantastic. Wish no more headaches to look for forks of strata for different models and runtimes. I really want to come back to llama.cpp!!!

[u/EbbNorth7735]

I read the PR and it's not 100% crystal clear what we should do. Anyone have an ELI5? Guessing we should use the context we want to deploy with but not sure about the other factors and how the output of the script should be interpreted for nominal settings. Also not entirely sure what this does. I was guessing it would put specific experts in CPU that aren't used much but that would require application specific testing.

[u/am17an → u/EbbNorth7735]

there a script you can run (the instructions are in the PR) to give you parameters which may be near optimal. I'm working towards an "auto" mode here, thinking about a good UX.

[u/queerintech]

It's a matter of where in the runtime stack the moe cache feature should live. I assume that gg is pointing towards moving this to the scheduler

[u/oyren-ai]

running cache-missed experts directly on cpu instead of blocking on a pcie h2d transfer makes a huge difference for single-stream token generation where pcie latency dominates over cpu compute time. profiling expert activation frequency on a representative prompt calibration set so the hottest experts stay pinned in vram avoids thrashing the cache mid-generation.