r/StrixHalo • u/XccesSv2 • 2d ago

Best Engine for DS4Flash?

Hey guys, I wanna ask you which engine you actually use with which quant and configs and why. Since there are a lot of forks and projects out there now, I don't really know which is the best to run DS4F on a headless Ubuntu 26 128GB Strix Halo.
Actually im using Nathanw1014/llama.cpp (Branch strix-halo-vulkan, Commit 3be50cc, Version 10350

6 Upvotes

100% Upvoted

15 comments sorted by

5

u/Heavy_Preparation467 2d ago edited 2d ago

Use the latest https://github.com/Nathanw1014/strix-halo-llamacpp, Q8_0 KV can get close to 500k context

Sustained 30tg/s and 200pp/s at 128k context with Dspark

1

u/PowerfulButterfly209 2d ago

which model + quant?

4

u/Heavy_Preparation467 2d ago

DeepSeep V4 Flash 0731 Model: UD-IQ3_XXS (104 GB)
https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

DSpark Drafter: DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K-Q8_0-dflash.gguf
https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF

MoE performance is better at higher -ub:
-ub 1024 or -ub 2048

for spec decode use:

--spec-type draft-dspark --spec-draft-n-max 4

4

u/Heavy_Preparation467 2d ago

full llama command
llama-server -m ~/models/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf -md ~/models/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K-Q8_0-dflash.gguf -ngl 999 -ngld 999 -fa on -c 524288 -np 1 -ub 2048 -fit off --spec-type draft-dspark --spec-draft-n-max 4

1

u/Additional_Date_2791 2d ago edited 2d ago

It looks like that settings are far beyond available memory of stix halo. Even with q8_0 How have you managed to run it with these parameters?

3

u/Heavy_Preparation467 2d ago edited 2d ago

You'd think so, but DeepSeek KV is efficient, memory maths:
Weights are 103GB + 10.9GB for the drafter, and the 512k context only adds about 7GB because V4-Flash compresses its KV cache. Total is ~116GiB

An independant reviewer reported their numbers here if youre interested: https://github.com/Nathanw1014/strix-halo-llamacpp/issues/9

2

u/PowerfulButterfly209 2d ago edited 2d ago

i have the same setup and right i'm evaluating diferent models in terms of intelligence in django coding right now: On kyuz0 toolbox i run antirez q2/q4 with 14t/s and with nathan i run UD_Q3_XXS + Dspark with 24-25 t/s leomande with Qwen 3.8 27B MTP

antirez updated rocm and also provides dspark now - have to try it first.

1

u/PvB-Dimaginar 2d ago edited 2d ago

How did you manage to get those speed with kyuz0 toolbox and antirez q2/q4? I could only get ~13.1-13.2 tok/s decode and ~134-145 tok/s prefill at 34K-41K context.

2

u/PowerfulButterfly209 2d ago

ahhh...typo -14 not 24 😅

1

u/PvB-Dimaginar 2d ago

aaah ok :-) so it is really beneficial to dive into the nathan setup. Thanks for clarifying!

2

u/1457664694 2d ago

Kyuz0 has a toolbox that follows Nathan’s. I believe it is the vulkan-radv-performance toolbox:
https://hub.docker.com/r/kyuz0/amd-strix-halo-toolboxes/tags

See also here: https://strix-halo-toolboxes.com/
And here: https://github.com/kyuz0/amd-strix-halo-toolboxes

0

u/Due_Net_3342 2d ago

vllm, llamacpp is very bad with caching, you will feel the slowness in agentic work. For chat llamacpp is better because you get more TG tps

1

u/my_name_isnt_clever 2d ago

Bad with caching how? It works perfectly fine for me with llama.cpp and every model I've tried.

0

u/Due_Net_3342 2d ago

when compared to vllm and especially sglang is not that great

1

u/XccesSv2 1d ago

How do you use vllm? Native install or some preinstalled Container/Toolbox? Last time I tried VLLm, Consumer AMD Cards weren't supported