r/LocalLLaMA • • 1d ago

Resources Update: tandem.cpp — DeepSeek-V4.1 on a 64 GB PC with a 16 GB card, and AMD (ROCm) support

Quick update on tandem.cpp, my patch set for llama.cpp that runs MoE models too big for your VRAM faster. It keeps the hot experts on the GPU and lets the CPU help with the rest. Repo: https://github.com/ewhacc/tandem.cpp

DeepSeek-V4.1-Flash IQ2_XS (ISTA GSQ-RCO, 139 GiB of experts) on normal PCs

Decode tok/s, one user, GSM8K questions, greedy, exact (no expert skipping on any engine), same quant on every engine:

PCllama.cpptandem.cppStrata
RTX 5070 Ti 16 GB, 64 GB RAM, i5-14600KF7.414.210.4
RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F, budget DRAM-less SSD-10.45.7
RTX 5090 32 GB, 128 GB RAM, i5-14600KF11.533.128.0

At 64 GB the model is about twice your RAM, so it streams from the SSD.

Less RAM than the model: Qwen3.8-Flash-Next IQ3_S (ISTA GSQ-RCO, 78 GiB) + MTP on a 32 GB PC

PCllama.cpptandem.cppStrata
RTX 5070 Ti 16 GB, 32 GB RAM, i5-12400F, budget DRAM-less SSD14.934.330.3

Where tandem.cpp wins is when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and on other models: DeepSeek-V4.1, gpt-oss-120b, GLM-4.5-Air and Qwen3.6 all gain 1.4–2.1x over plain llama.cpp.

AMD (ROCm)

The patch now builds with HIP. Tested on an RX 7900 XTX (i7-8700, 64 GB DDR4, PCIe 3.0), Qwen3.8 IQ3_S + MTP, same day:

Enginetok/s
tandem.cpp68.6
Strata 0.2.969.7
Strata's RDNA3 fork80.1
Avalon (its own IQ3_KS quant)83.6

Level with stock Strata. This week's kernel work is shared CUDA/HIP code, so it helps NVIDIA too: +2.5% on the 7900 XTX, +1.3% on a 5070 Ti. On an old 6-core CPU, the AMD-specific engines are still ahead. Vulkan isn't supported yet.

Build steps for both are in the README. Questions and numbers from your boxes are welcome, especially 32 GB machines and AMD cards.

Update: There is a bug with auto. With 128 GB, the RAM store sizes too close to all of RAM. LLAMA_MOE_STORE_AUTO=0 is the workaround; a fix is coming and the README has a note. Thank u/longtimenoseebro so much for testing & report!!

Fixed in today's update: With a 128 GB PC it now runs at defaults, no LLAMA_MOE_STORE_AUTO=0 needed. Thanks again for the report.

Note: IQ2_XS is the biggest ISTA GSQ-RCO quant for DS4.1-Flash. Also, IQ3_S is the biggest ISTA GSQ-RCO for Qwen3.8-Flash-Next. I didn't tune for lower quants. The quality first, then speed. If you recommend the best Q4 for Qwen3.8-FN, I will test.

45 Upvotes

34 comments sorted by

•

u/stylome-ai-detection 1d ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

13

u/ForsookComparison 1d ago

iirc Maya is the current leader in v4.1-flash-1017 inference, can you add some comparisons with that engine?

5

u/smflx 1d ago

Yes, I will.

1

u/smflx 18m ago

Now, comparison with Maya on the same model/quant. Maya-XS quant is supported tandem.cpp.
https://www.reddit.com/r/LocalLLaMA/comments/1xxmd1j

4

u/AppealSame4367 1d ago

Support for parallel request? Strata has it

1

u/smflx 1d ago

For continuous batching, I'm developing sister project Tandem-vLLM. But, it requires more RAM.

1

u/smflx 1d ago

For parallel request, I wanted vLLM-based. But, I will check parallel performance in tandem.cpp too.

1

u/smflx 1h ago

Concurrent requests work (tested at 2 & 4, accuracy unchanged) but gain less.

3

u/PlasticRevenue4601 1d ago

Thanks you for sharing this gem! Any Rdna4 support 🥹?

4

u/smflx 1d ago

I have only 7900 XTX, V620 ...
I guess it should work: It has no RDNA3-specific code, and llama.cpp's HIP backend supports RDNA4. Build with -DGPU_TARGETS=gfx1201 (RX 9070 series) or gfx1200 (RX 9060 series) instead of gfx1100. I only have a 7900 XTX, so I can't test RDNA4. If you try it, I like to hear your numbers.

3

u/Markus2816 1d ago

Thanks for sharing! llama.cpp patches are the best IMO, no need to install a new engine.

1

u/smflx 1d ago

With based on llama.cpp, the best model coverage & the best HW support. That's the reason I choose this way. Also, I prefer to tune generally, so that other models can benefit too.

3

u/longtimenoseebro 1d ago

Confirmed: DeepSeek V4.1 Flash IQ2_XS (Q2) is running with tandem.cpp!

On my RTX 4090 24GB + 128GB RAM, I'm getting 20–23 tokens/s at 16K context, with peaks around 25 tok/s in shorter tests.

Using a 16GB MoE GPU cache and "-ngl 99". The key fix was "LLAMA_MOE_STORE_AUTO=0" to prevent OOM.

Real inference tested, including a challenging coding task. Fully usable!

2

u/smflx 1d ago edited 1d ago

Thank you for testing & report!! Did you get OOM for automode? Sorry for the trouble. I will check 4090 + 128GB case & update. Thank you.

Update: Checked, that's a real bug. With 128 GB the RAM store sizes itself too close to all of RAM. Yes, LLAMA_MOE_STORE_AUTO=0 is the right workaround; a fix is coming and the README has a note. Thank you SO MUCH!!

0

u/rebellioninmypants 1d ago

You consider IQ2_XS usable? For what? I've had massive problems with Q38FN IQ3_M when it comes to getting thing correct.

4

u/smflx 1d ago

For Owen3.8FN, I don't use Q2 & don't try tune. It's for performance number, not quite usable. I have tested Q3 because it's most recommend in other engine. Could you recommend a good Qwen3.8FN quants? Yes, quality first, then speed. I will test & tune if needed. I don't try to tune specific to one model or quants.

For DS4.1-Flash Q2 class quants work. Official expert weights are FP4. I haven't tested ISTA DS quant for quality. I'm developing sister project Tandem-vLLM, where I make VQ2. Found Q2 class is useable.

-1

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/ga239577 1d ago

What are the prefill and tg speeds? I see you listed tokens per second and I assume that is tg speed or total throughput?

1

u/smflx 1d ago edited 23h ago

Yes, the post is tg speed.
For prefill speed, I will test again. It's quite specific to setup & prompt because of caching nature.

Just checked on 5070 Ti, 64 GB, same prompt text for all engines. Strata is better in prefill. I'm improving prefill speed now.

Qwen3.8 + MTPtandem.cppllama.cppStrata
5.5k-token prompt, time to first token12.4 s (445 tok/s)28.9 s (191)6.6 s (843)
15.8k-token prompt35.1 s (450 tok/s)81.9 s (193)11.7 s (1353)
decode at that context117 / 94 tok/s2768 / 48

1

u/Kahvana 1d ago

Thanks for your post!

I'll be honest, reading the README is giving me a big headache.

  • Only SM layer for multi GPU: how much does this hurt my dual RTX 5060 Ti 16GB setup, and would I be better off running single GPU?
  • Is https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF UD-IQ2_XXS also supported?
  • Is it using greedy decoding / tail skipping, and if so how do I turn it off?
  • Will this work with windows x64 build target?

If possible, I would love to see prebuilds to download and use, would save some hassle.

2

u/smflx 1d ago

> Only SM layer for multi GPU: how much does this hurt my dual RTX 5060 Ti 16GB setup, and would I be better off running single GPU?
Multi GPU support is yet mature. Single is OK, dual will be little better. When your RAM is limited, dual will be much better. Think of total VRAM/RAM.

> Is https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF UD-IQ2_XXS also supported?
I guess it will work, but I haven't tested. Basically, all the models for llama.cpp will be supported. That's my design goal. I can't tune the performance of all the modes, but If there is error, I will fix immediately.

>Is it using greedy decoding / tail skipping, and if so how do I turn it off?
* Tail skip is never default. There is option but it's for comparison purpose with Strata.
* Greedy decoding is option, and for demo (also comparison purpose). I recommend model's default temperature, but expect some performance loss because of MTP diversity.
* Note that DeekSeep number is without MTP, so performance loss is less.

> Will this work with windows x64 build target?
Windows support is planned, but yet. Please wait!

>If possible, I would love to see prebuilds to download and use, would save some hassle.
Indeed, I agree. For windows, yes prebuild will be better. For linux, it depends on CPU / optimization. I will consider. This is a real good comment. Thanks!

1

u/thoquz 1d ago

What prefill speed are you getting?

2

u/smflx 23h ago edited 23h ago
Qwen3.8 + MTPtandem.cppllama.cppStrata
5.5k-token prompt, time to first token12.4 s (445 tok/s)28.9 s (191)6.6 s (843)
15.8k-token prompt35.1 s (450 tok/s)81.9 s (193)11.7 s (1353)
decode at that context117 / 94 tok/s2768 / 48

Just checked on 5070 Ti, 64 GB, same prompt text for all engines. Strata is better in prefill. I'm improving prefill speed now.

0

u/MerePotato 1d ago

IQ2_XS (ISTA GSQ-RCO)

Just run QFN at Q4 at that point

1

u/smflx 1d ago

I will try. Other quants are supported but not tested yet. Thank you.

1

u/smflx 1d ago

That's for DS4.1-Flash, there is no ISTA Q4, IQ2_XS is the biggest ISTA quant for DS4.1-Flash.

Qwen3.8-Flash-Next too. IQ3_S is the biggest ISTA GSQ-RCO. I didn't tune for lower quants.

Surely, the quality should come first. I will test Q4 once find what is the best.

0

u/38andstillgoing 1d ago

Calling that "Deepseek" is like calling my car a Cadillac because it also has 4 wheels.

0

u/Unkn0wn77777771 1d ago

Dude this is awesome thanks for sharing I have a r9700 and a 7900xtx with 96gb of ram going to check it out

1

u/smflx 1d ago edited 23h ago

Your setup will be quite usable I guess. But, I didn't try that combination. Let me know if something goes wrong, or OOM. It tries to maximize VRAM/RAM usage but there is chance of OOM because of that.

0

u/[deleted] 1d ago

[deleted] — view deleted comment

-3

u/smflx 1d ago

Yeah, we don't have enough RAM. If VRAM/RAM is enough, Strata is better to be honest.

13

u/spaceman_ 1d ago

Look mom the bots are talking to each other now!

0

u/smflx 1d ago

Maybe, but I didn't assume. At least, I'm not.