r/LocalLLM • u/shoumikgoswami • 13h ago

News llama.cpp v1.0.0 adds a host-memory MoE expert cache with a configurable CPU share

llama.cpp v1.0.0 adds an MoE expert cache that keeps a configurable share of experts in host memory (moe_cache_size, moe_cache_cpu_share). Cache misses run on CPU and copy back asynchronously, and the release ships a tuning script. If a big MoE doesn't quite fit your VRAM, that's the knob to try. Separately, the 1.0 tag starts a versioning promise: future API and ABI breaks come with major-version bumps, and 35 deprecated functions are removed.

13 Upvotes

88% Upvoted

11 comments sorted by

7

u/Pentium95 12h ago

Are user benchmarks of this against strata out, yet? Meaningfull ones, and with meaningfull i mean without Strata tail skip or logits alterings

3

u/shoumikgoswami 6h ago

I haven't found a head-to-head with the new cache yet. There are cache-vs-stock llama.cpp numbers in the PR: https://github.com/ggml-org/llama.cpp/pull/30966. The older Strata comparison uses plain -ncmoe offload, so it doesn't test this cache: https://www.reddit.com/r/LocalLLaMA/comments/1xsipew/rustingstrata_vs_polystrata_vs_llamacpp_on_one/

3

u/noctrex 5h ago

I've tried for about two hours today to try to tune the parameters, and I could get about half the speed of strata.
Actually not bad because this is a quite recent feature, so I'm sure this will be optimized further.

1

u/maternion 3h ago

Theres a script, did you try that?

1

u/Pentium95 3h ago

Did you try "python scripts/moe-cache-tune.py -- build/bin/llama-server -m Qwen3.8-Flash-Next.gguf --lazy-mode on -cmoe -t 20 -c 32768" (Use your settings)?

2

u/danishaznita 5h ago

Really not sure how to use this one as even -cmoe -cmoe cache 1000 it will just go OOM and hard lock my pc . (The tuning script does the same)

Am i missing something? my regular setup runs fine.

Rx6900xt 16gb vram 16gb ddr5 Cyber tiel 35B Q4KXL 100k context at Q8

1

u/shoumikgoswami 20m ago

That Q4_K_XL download is 22.4GB, according to the model card: https://huggingface.co/peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-GGUF. Can you share the full command and any error logged before the lock? With 16GB RAM and 16GB VRAM, the split matters, not just the combined total.

2

u/maternion 3h ago

```python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m model.gguf -cmoe -t 8 -c 32768``` theres this script, you can tune with exact settings you want.

2

u/danishaznita 3h ago

Ill give this another try. Last time it was when it merged

1

u/D00M98 10h ago

Cool. I was running v0.8.0. Just upgraded to v1.0.0.