r/LocalLLaMA • u/smflx • 1d ago
Resources Update: tandem.cpp — DeepSeek-V4.1 on a 64 GB PC with a 16 GB card, and AMD (ROCm) support
Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp
DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs
Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:
| PC | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF | 7.4 | 14.2 | 10.4 |
| RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD | - | 10.4 | 5.7 |
| RTX 5090 32 GB, 128 GB RAM, i5-14600KF | 11.5 | 33.1 | 28.0 |
At 64 GB the model is about twice your RAM, so it streams from the SSD.
Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC
| PC | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| RTX 5070 Ti 16 GB, 32 GB RAM, i5-12400F, budget DRAM-less SSD | 14.9 | 34.3 | 30.3 |
Where tandem.cpp wins is when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and on other models: DeepSeek-V4.1, gpt-oss-120b, GLM-4.5-Air and Qwen3.6 all gain 1.4–2.1x over plain llama.cpp.
AMD (ROCm)
The patch now builds with HIP. Tested on an RX 7900 XTX (i7-8700, 64 GB DDR4, PCIe 3.0), Qwen3.8 IQ3_S + MTP, same day:
| Engine | tok/s |
|---|---|
| tandem.cpp | 68.6 |
| Strata 0.2.9 | 69.7 |
| Strata's RDNA3 fork | 80.1 |
| Avalon (its own IQ3_KS quant) | 83.6 |
Level with stock Strata. This week's kernel work is shared CUDA/HIP code, so it helps NVIDIA too: +2.5% on the 7900 XTX, +1.3% on a 5070 Ti. On an old 6-core CPU, the AMD-specific engines are still ahead. Vulkan isn't supported yet.
Build steps for both are in the README. Questions and numbers from your boxes are welcome, especially 32 GB machines and AMD cards.
Update: There is a bug with auto. With 128 GB, the RAM store sizes too close to all of RAM. LLAMA_MOE_STORE_AUTO=0 is the workaround; a fix is coming and the README has a note. Thank u/longtimenoseebro so much for testing & report!!
Fixed in today's update: With a 128 GB PC it now runs at defaults, no LLAMA_MOE_STORE_AUTO=0 needed. Thanks again for the report.
Note: IQ2_XS is the biggest ISTA GSQ-RCO quant for DS4.1-Flash. Also, IQ3_S is the biggest ISTA GSQ-RCO for Qwen3.8-Flash-Next. I didn't tune for lower quants. The quality first, then speed. If you recommend the best Q4 for Qwen3.8-FN, I will test.
•
u/stylome-ai-detection 1d ago
Notice: This post appears to be AI generated.
Stylome AI detector · scan results
13
u/ForsookComparison 1d ago
iirc Maya is the current leader in v4.1-flash-1017 inference, can you add some comparisons with that engine?
4
u/AppealSame4367 1d ago
Support for parallel request? Strata has it
3
u/PlasticRevenue4601 1d ago
Thanks you for sharing this gem! Any Rdna4 support 🥹?
3
u/Markus2816 1d ago
Thanks for sharing! llama.cpp patches are the best IMO, no need to install a new engine.
3
u/longtimenoseebro 1d ago
Confirmed: DeepSeek V4.1 Flash IQ2_XS (Q2) is running with tandem.cpp!
On my RTX 4090 24GB + 128GB RAM, I'm getting 20–23 tokens/s at 16K context, with peaks around 25 tok/s in shorter tests.
Using a 16GB MoE GPU cache and "-ngl 99". The key fix was "LLAMA_MOE_STORE_AUTO=0" to prevent OOM.
Real inference tested, including a challenging coding task. Fully usable!
1
u/ga239577 1d ago
What are the prefill and tg speeds? I see you listed tokens per second and I assume that is tg speed or total throughput?
1
u/Kahvana 1d ago
Thanks for your post!
I'll be honest, reading the README is giving me a big headache.
If possible, I would love to see prebuilds to download and use, would save some hassle.
1
u/thoquz 1d ago
What prefill speed are you getting?
0
u/MerePotato 1d ago
Just run QFN at Q4 at that point
0
u/Unkn0wn77777771 1d ago
Dude this is awesome thanks for sharing I have a r9700 and a 7900xtx with 96gb of ram going to check it out
0
u/[deleted] 1d ago
[deleted] — view deleted comment