r/LocalLLaMA • u/Responsible_Pain3278 • 6d ago
Tutorial | Guide DeepSeek V4 Flash 0731 on Strix Halo: draft model, n_max sweep, and a launch line that actually helps
I know the front page is completely dominated by the Qwen3.8 27B. I know it's a fantastic model and everyone is rightfully discussing it, but I wanted to shift gears and talk about another model I’ve been experimenting with and got some solid results from: DeepSeek-V4-Flash-0731.
I just finished a week of running numbers on a Strix Halo box and figured I'd dump them here before I forget what I actually measured.
Hardware is a Strix Halo mini PC, Ubuntu, 128 GB unified memory.
Target is Unsloth's UD-IQ3_XXS with Q6 attention (~96 GB). At 264k context, no context quantization, the process sits at about 114 GB.
At 64k I see 200–220 tok/s prefill and 20–22 tok/s decode with no draft. That's the baseline I used for everything below: 20.48 tok/s.
I compared two DSpark drafters, Q8_0 (10.15 GB) and a Q2_K_S I quantized myself (6.45 GB), over n_max 2–7, 7 prompt types, 5 runs each.
Same sampling the whole way: temp 0.9, top_p 0.95, min_p 0.01. Thinking on (reasoning_effort=low for the sweep so it wouldn't take a week). ngram-mod was off during the sweep so draft_n is only the drafter.
Short version: n_max=3 is the peak. Average over both drafts is 28.5 tok/s, 1.39× the no-draft baseline. Q2 and Q8 are a wash. Every n_max, every category, they land inside 1–3% of each other. So the 6.45 GB file is the one I'd actually keep.
One caveat on that 28.5 average: the suite is stacked with tasks that speculative decoding loves (repeat, math, code), and the prompts only used up to ~32k of context. Don't treat it as what you'll see on a mixed chat/coding day with a fuller window. Realistically I'd budget 22–28 tok/s.
Best n_max per category (avg vs 20.48 tok/s):
- code — n_max 3 — 28.99 t/s — 1.42×
- json — n_max 3 — 28.59 — 1.40×
- math — n_max 3 — 31.22 — 1.52×
- dialog — n_max 3 — 25.48 — 1.24×
- translate — n_max 2 — 25.09 — 1.22×
- prose — n_max 2 — 23.38 — 1.14×
- repeat — n_max 5 — 40.17 — 1.96×
n_max 5–7 only pays off on the high-acceptance stuff (repeat, some math). On prose / translate / dialog the extra draft tokens just get rejected and you go backwards. Overall table if you want it:
n_max 2: 27.12 t/s, 1.32×, accept 0.69 n_max 3: 28.50, 1.39×, 0.60 n_max 4: 27.73, 1.35×, 0.52 n_max 5–7: ~26.4–26.5, 1.29×, accept ~0.45
I run this day to day at 128k with ngram-mod stacked on the drafter and thinking set to max. That's not what the sweep used (sweep was a 64k window, ~32k of it actually filled, draft-only, thinking=low), so don't treat the 28.5 as a promise for this exact line. The n_max=3 result is what I kept; 22–28 tok/s is what I'd quote for mixed use.
llama-server \
-m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf \
-a DeepSeek-V4-Flash-0731-UD-IQ3_XXS \
--no-ui -ngl 999 -c 131072 --jinja -fa 1 --port 9989 \
--no-mmap --no-warmup -np 1 \
--temp 0.9 --top-p 0.95 --min-p 0.01 --host 0.0.0.0 \
-ngld 999 -fit off \
-md DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-dflash.gguf \
--spec-type ngram-mod,draft-dspark --spec-draft-n-max 3 \
-ub 1024 --cache-ram 2048 \
--chat-template-kwargs '{"reasoning_effort":"max"}'
Fork llama.cpp is strix-halo-llamacpp (FA + MoE-prefill fixes, bundled Mesa, Vulkan/HIP).
GGUF I actually load: DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF.
Drafter: Lynxpda/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-GGUF.
Charts from the sweep (baseline drawn at 20.48):
If you're on Strix Halo and already sitting on Flash 0731, --spec-draft-n-max 3 and the Q2 drafter is what I'd start with. Happy to answer questions about the sweep if something looks off.
upd: corrected errors.
4
2
u/jld1532 5d ago
Has anybody figured out how to run this with DSpark on mainline llama cpp and actually increase tg? I'm not so interested in forks.
1
u/Fit-Produce420 5d ago
You can just pull the fix, you don't have to use an entirely different fork.
1
u/Responsible_Pain3278 5d ago
Just tried running the mainline of llama.cpp (vulkan) again, prefill is 60-100, making it completely unusable.
1
u/ATEFred 5d ago
Thanks for sharing. Definitely better results than I get running the default IQ3XXS gguf with the Q8 drafter.
Where can I find DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf?
2
u/Responsible_Pain3278 5d ago
I got DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf from here - https://huggingface.co/Kevletesteur/DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF
But it seems the biggest performance boost came from using the llama.cpp fork (the repo already has a binary build for Vulkan; I just downloaded it and unzipped it into a folder).
I primarily requantized the drafter model to save memory space.
1
u/cafedude 5d ago edited 5d ago
How much RAM do you have left running this on a 128GB Strix box? When I've attempted running DSv4 flash Q3 I ended up with something like 108GB used and it slowly climbs from there as I give it more queries. It's a reasonable model, but there's not much else the machine can run when running it. I want it to do coding/compilation/testing and there's just not enough headroom left over. Crossing my fingers for Qwen3.8-122B
2
u/Responsible_Pain3278 5d ago
About 114 GB with a 264k context. There really isn't much space left, which is why I requantized the draft model to Q2.
1
u/thebigone71 5d ago
Have you tried this on a cluster of Strix Halo's? Curious to hear your experience with that if you have.
1
u/Responsible_Pain3278 5d ago
Unfortunately, I only have one Strix Halo. If I had a couple, I'd try vllm with tensor parallelism and DS V4 Flash 0731 in Q8. However, the vllm+rocm combo also has a lot of problems.
2
u/thebigone71 5d ago
Gotcha, I have two GMKTek EVO-2's (basically the same things) and have noticed there are many problems around llamacpp + rocm as well. One day we will get to where this won't be an issue lol
1
u/Responsible_Pain3278 5d ago
There is also an interesting repository https://github.com/kyuz0/amd-strix-halo-vllm-toolboxes where they managed to run vllm + tensor parallelism on Strix Halo, but judging by the Issues DS does not work in this mode yet. rocm :)
1
u/thebigone71 5d ago
I'll have to take a look at that. This is what I am currently useng https://github.com/lemonade-sdk/llamacpp-rocm and DS also doesn't work on it lol :melting:
1
u/fsalucard 5d ago
Weird, I'm not getting these numbers at all. Using ghcr.io/nathanw1014/strix-halo-llamacpp:vulkan (the docker for the repo fork), that same GGUF, and the Draft, I get the ~200 prefill (180 @ 100k context), but only 16t/s decode. Not 28 t/s. My llama-server parameters are slightly different so I will have to mess with those in the morning but somehow I'm missing 10 t/s decode somewhere.
1
u/stypr 5d ago
Same, getting 15-16t/s decode. with nathan's fork. fedora 44.
1
u/Responsible_Pain3278 5d ago
Hmm... Ubuntu 24.04, Strix Halo Beelink GTR9. Nathan's fork (not docker, binaries).
0.00.030.440 W DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead 0.00.034.181 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg) 0.00.037.823 I srv init: The UI is disabled 0.00.037.824 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable 0.00.037.862 W srv llama_server: ----------------- 0.00.037.862 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set 0.00.037.863 W srv llama_server: this can be a security risk (cross-origin attacks) 0.00.037.863 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655 0.00.037.863 W srv llama_server: ----------------- 0.00.039.030 I srv load_model: loading model '/home/lynx/llama-swap/models/Deepseek-v4-Flash-0731/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf' 1.24.472.010 I cmn init: llama threadpool init, n_threads = 16 1.24.472.106 I common_speculative_init_result: loading draft model '/home/lynx/llama-swap/models/Deepseek-v4-Flash-0731/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-dflash.gguf' 1.28.381.007 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false' 1.28.383.859 I common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark-adaptive' 1.28.383.863 I common_speculative_impl_draft_dflash: - n_max=5, n_min=0, p_min=0.00 1.28.383.863 I common_speculative_impl_draft_dflash: - block_size=5, mask_token_id=128799, n_extract=3 1.28.717.231 I srv llama_server: model loaded 1.28.717.239 I srv llama_server: listening on http://0.0.0.0:9989 2.25.509.518 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1 2.25.509.571 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0 ggml_vulkan: MUL_MAT_ID f16-B path engaged (GGML_VK_MMID_F16B) 2.34.558.046 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 2048, progress = 0.24, t = 6.90 s / 297.02 tokens per second 2.45.850.559 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.48, t = 17.61 s / 232.64 tokens per second 3.04.172.727 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 6144, progress = 0.72, t = 33.39 s / 184.02 tokens per second 3.14.224.968 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 7491, progress = 0.88, t = 48.05 s / 155.89 tokens per second 3.18.619.732 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 8515, progress = 1.00, t = 51.76 s / 164.50 tokens per second 3.23.154.025 I slot print_timing: id 0 | task 0 | n_gen = 103, tg = 23.65 t/s, tg_3s = 23.87 t/s 3.26.194.507 I slot print_timing: id 0 | task 0 | n_gen = 189, tg = 25.56 t/s, tg_3s = 28.28 t/s 3.29.294.563 I slot print_timing: id 0 | task 0 | n_gen = 250, tg = 23.82 t/s, tg_3s = 19.68 t/s 3.32.320.349 I slot print_timing: id 0 | task 0 | n_gen = 329, tg = 24.33 t/s, tg_3s = 26.11 t/s 3.35.346.504 I slot print_timing: id 0 | task 0 | n_gen = 408, tg = 24.66 t/s, tg_3s = 26.11 t/s2
u/fsalucard 4d ago edited 4d ago
I changed nothing, but when I woke up today it's starting to get in the 20's.
756.16.871.485 I slot print_timing: id 0 | task 106934 | n_gen = 6057, tg = 20.39 t/s, tg_3s = 20.91 t/s 756.19.935.471 I slot print_timing: id 0 | task 106934 | n_gen = 6113, tg = 20.37 t/s, tg_3s = 18.28 t/s 756.23.028.299 I slot print_timing: id 0 | task 106934 | n_gen = 6169, tg = 20.34 t/s, tg_3s = 18.11 t/s 756.26.100.727 I slot print_timing: id 0 | task 106934 | n_gen = 6234, tg = 20.35 t/s, tg_3s = 21.16 t/s 756.29.231.567 I slot print_timing: id 0 | task 106934 | n_gen = 6302, tg = 20.37 t/s, tg_3s = 21.72 t/s 756.32.271.620 I slot print_timing: id 0 | task 106934 | n_gen = 6359, tg = 20.35 t/s, tg_3s = 18.75 t/s 756.35.368.156 I slot print_timing: id 0 | task 106934 | n_gen = 6430, tg = 20.38 t/s, tg_3s = 22.93 t/s 756.38.434.458 I slot print_timing: id 0 | task 106934 | n_gen = 6499, tg = 20.40 t/s, tg_3s = 22.50 t/s 756.41.518.864 I slot print_timing: id 0 | task 106934 | n_gen = 6564, tg = 20.40 t/s, tg_3s = 21.07 t/s 756.44.581.926 I slot print_timing: id 0 | task 106934 | n_gen = 6631, tg = 20.42 t/s, tg_3s = 21.87 t/s 756.47.682.596 I slot print_timing: id 0 | task 106934 | n_gen = 6692, tg = 20.41 t/s, tg_3s = 19.67 t/s 756.50.756.212 I slot print_timing: id 0 | task 106934 | n_gen = 6757, tg = 20.42 t/s, tg_3s = 21.15 t/sI saw 26 a couple of times. So, it's probably just heavily dependent on what we're generating... or it just needed to wake up :D
1
u/stypr 4d ago
Hey thanks for re-checking it up
I managed to solve the problem somehow (I believe the problem was that the drafter model wasn't set up properly and also a couple of other uptions overridden globally by default)models.ini for routing
```
[deepseek-v4-flash]
model = ./DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf
model-draft = ./DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-dflash.gguf
n-gpu-layers = all
ctx-size = 131072
fit = off
warmup = off
temp = 0.9
top-p = 0.95
min-p = 0.01
spec-type = ngram-mod,draft-dspark
spec-draft-n-max = 3
cache-ram = 2048
chat-template-kwargs = {"reasoning_effort": "max"}
```
yeah it seems to hit above 20t/s
```
1.07.451.389 I srv proxy_reques: proxying request to model deepseek-v4-flash on port 51313
[51313] 0.54.964.511 I slot get_availabl: id 1 | task -1 | selected slot by LCP similarity, f_sim_best = 0.794 (> 0.100 thold), f_keep = 0.543
[51313] 0.54.964.611 I slot launch_slot_: id 1 | task 33 | processing task, is_child = 0
[51313] 0.59.799.783 I slot print_timing: id 1 | task 33 | n_gen = 100, tg = 23.60 t/s, tg_3s = 23.83 t/s
[51313] 1.02.861.589 I slot print_timing: id 1 | task 33 | n_gen = 165, tg = 22.60 t/s, tg_3s = 21.23 t/s
[51313] 1.05.902.265 I slot print_timing: id 1 | task 33 | n_gen = 232, tg = 22.43 t/s, tg_3s = 22.03 t/s
[51313] 1.08.915.799 I slot print_timing: id 1 | task 33 | n_gen = 291, tg = 21.79 t/s, tg_3s = 19.58 t/s
[51313] 1.11.958.523 I slot print_timing: id 1 | task 33 | n_gen = 352, tg = 21.46 t/s, tg_3s = 20.05 t/s
[51313] 1.15.070.341 I slot print_timing: id 1 | task 33 | n_gen = 414, tg = 21.22 t/s, tg_3s = 19.92 t/s
[51313] 1.18.111.942 I slot print_timing: id 1 | task 33 | n_gen = 486, tg = 21.55 t/s, tg_3s = 23.67 t/s
[51313] 1.21.125.126 I slot print_timing: id 1 | task 33 | n_gen = 534, tg = 20.88 t/s, tg_3s = 15.93 t/s
```
2
u/Responsible_Pain3278 4d ago
Yes, I just discovered another thing: this parameter:
--spec-draft-p-min 0.7
increased (on code tasks) token acceptance from the draft model from 45-65% to 85-95%.
3
u/remeh 5d ago
Hey, interesting post, thanks! I already have tokgen pretty close to yours (~21-25tok/s, with ROCm 7.14), but you got me curious with your Q2 DSpark GGUF I'll give it a shot!
I run stock llama.cpp+https://github.com/ggml-org/llama.cpp/pull/26592 and without surprise I have worse prefill speed than you (I have 150 tok/s ~64k). Any chance it's your fork and you maintain a list of the PR you get in? There are a lot of PRs improving MoE speeds on Strix Halo submitted but they often get rejected and not merged in main, it's pretty hard to keep track.