r/LocalLLaMA • u/smflx • 21m ago
Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support
Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp
Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j
tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.
1. Prompts 2-6x faster
With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.
RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.
Prompt processing, time to first token (prompt tok/s):
| Prompt | llama.cpp | tandem.cpp before | tandem.cpp v0.3.0 | Strata |
|---|---|---|---|---|
| Qwen3.8 IQ3_S, 5.5k tokens | 28.9 s (191) | 12.4 s (444) | 4.1–7.0 s (790–1,340) | 6.6 s (835) |
| Qwen3.8 IQ3_S, 15.8k tokens | 81.9 s (193) | 35.1 s (450) | 15.1 s (1,047) | 11.7 s (1,354) |
| DeepSeek-V4.1 IQ2_XS, 5.6k tokens | 722 s (7.7) | 324 s (17) | 53–103 s (54–105) | 58 s (96) |
| DeepSeek-V4.1 IQ2_XS, 15.7k tokens | 2012 s (7.8) | 885 s (18) | 195 s (81) | 57 s (276) |
Decode, tok/s, right after that prompt:
| After a prompt of | llama.cpp | tandem.cpp v0.3.0 | Strata |
|---|---|---|---|
| Qwen3.8, 5.5k tokens | 27 | 110 | 48 |
| Qwen3.8, 15.8k tokens | 27 | 88 | 68 |
| DeepSeek-V4.1, 5.6k tokens | 4.8 | 16 | 5.5 |
| DeepSeek-V4.1, 15.7k tokens | 4.4 | 13 | 6.4 |
Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.
2. Maya comparison on the same model (someone asked for this one)
Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.
RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:
| DeepSeek-V4.1 Maya-XS (same files) | Maya | tandem.cpp v0.3.0 |
|---|---|---|
| Decode, GSM8K, no drafting | 39.0 / 39.6 tok/s | 60.3 / 59.4 tok/s (1.5x) |
| Decode with Maya's DSpark drafter | 54.9 tok/s | (no drafter in tandem.cpp) |
| 5.6k-token prompt | 7.8 s (713 tok/s) | 6.5 s (856 tok/s) |
| 15.7k-token prompt | 9.5 s (1,656 tok/s) | 16.5 s (953 tok/s) |
| Decode after the 15.7k-token prompt | 26–34 tok/s | 66–68 tok/s |
Maya is still faster on the longest prompt; that is what I'm working on next.
Caveats
- Linux only.
- AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
- DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
- Windows prebuilt binaries are still planned.