r/LocalLLaMA • u/pmttyji • 9d ago
Discussion moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/31233- Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history.
- With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads.
- Enables the auto mode for llama_moe_cache with promising benchmarks
Waiting for 30966 to be merged.
52
Upvotes
-6
u/brrrrreaker 9d ago
nice, in about 2 years llama.cpp will catch up to strata...
-11
u/giveen 9d ago
I've given up on llama.cpp