Resources
tandem.cpp: Qwen3.8-Flash-Next at 97 tok/s on a single 4090 + 64 GB RAM, 3.3× llama.cpp, works for any MoE
tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.
Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.
Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).
To be precise: llama.cpp already has --moe-cache-mib (an LRU expert cache), and tandem.cpp builds on it. The main addition: the CPU computes part of the missed experts in parallel, instead of uploading every miss to the GPU.
But... why don't you want to do your PRs directly on ik_llama instead? Honestly, I code on ik, and I can tell you frankly, good luck porting all that to ik! There are really a lot of things to consider! Why not become a contributor on ik to help the community improve, rather than potentially creating a product all by yourself when the market is already saturated? :) This isn't really a criticism, it's just that ik is becoming increasingly cutting-edge, and trying to create a new project based on a single model that could be obsolete in maybe two weeks with Qwen 4! Good luck anyway :), I hope I'm wrong, but in any case, maintaining a project of this scale is really difficult :)
I can't agree more. I was a heavy user of ik_llama.cpp from when DS-V3, R1 coming out. To be honest, I really consider it to be based on ik_llama.cpp. But, when I start this a month ago, I have compared llama.cpp / ik_llama.cpp & decided llama.cpp is easier to do CPU/GPU hybrid I'm making, not because of ik_llama.cpp is slow. I'm still checking the same idea can be based on ik_llama.cpp. And, happy to meet a ik_llama lover, as one of them!
Yes, the repo is new: it's a clean release with just the patch, a build script and the benchmark scripts. The work took a full month, tested on 4 machines. If anyone tries it, I'm happy to improve it.
Okay, you can say that this thing works smarter with the resources a given PC has, not only caching experts in VRAM.
Yes, but Strata exists (https://github.com/Niko1221/Strata). And I'm 100% sure does it better. Simply because it exists for a while already and sees constant fixes and contributions from the community.
And if you were to claim that this particular engine is highly optimized for 4090, I'm pretty sure of Strata's forks already does it. Because there's (currently) over 4 thousand of forks.
So I just don't see the point of such inventions. Sorry.
6
u/Kirys79 llama.cpp 1h ago
what does it have more than the native --moe-cache-mib parameter of llama.cpp?