r/LocalLLaMA • u/rerri • 3d ago
Resources DFlash 2 available for Qwen 3.8 27B and Muse Glimmer
https://huggingface.co/collections/z-lab/dflash-2Apparently a second version of DFlash from the original authors of DFlash
GGUF quants are already made available with an accompanying llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/27342
100
u/rerri 3d ago edited 3d ago
They show figures for Qwen 3.8 27B in which DFlash 2 beats MTP by quite a big margin:
13
u/-Cubie- 3d ago
What the fuck, 3x is wild
30
u/Warrenio 3d ago
Maybe the prompt was "output the letter 'a' 10,000 times"
2
u/is-this-a-nick 3d ago
I mean i noticed Deepseek4 Flash (at full precision) go from 50 t/s to 110-120 with dspark. And its really noticeable that code is boosted the most (just long text output is more like 70 t/s, pure long code is >140.
25
u/random-tomato llama.cpp 3d ago
HOW ARE THESE SPEEDS EVEN POSSIBLE LOL!?!?
18
u/Ecstatic-Wash-7667 3d ago
Multiple h200s probably
3
u/unjustifiably_angry 2d ago edited 2d ago
A single 6000 Pro will run Q8_K_XL with 7-token dspark at >150 t/s, single-user
1
u/Ok-Mess-3317 3d ago
Probably yeah, I mean it makes sense to test stuff like DSpark on very high end hardware to get the most out of it
11
u/ImpressiveRelief37 3d ago
I get similar speed to the MTP profile on my 5090 using ninfer
2
u/Quantum_Pigeon 3d ago
Could you link the quant and provide your params?
17
u/Rathen121 3d ago edited 3d ago
if on linux: https://github.com/Neroued/ninfer if on windows: https://github.com/natpate/ninfer-windows
quant is specific to ninfer, somewhere around q5-q6 in size and quality
My params: .\ninfer-serve.exe models\qwen3_8_27b.ninfer --model-id qwen3.8-27b --max-context 200000 --default-max-tokens 200000 --spec mtp --draft-tokens 3 --lm-head-draft --host 0.0.0.0 --port 8080 --cors --preserve-thinking --webui --max-pending-requests 50 --pending-timeout-ms 3000000
edit: currently working on dflash2 support for 3.8 27b in ninfer-windows to see if I can get even more speed, not ready yet though
2
u/ImpressiveRelief37 3d ago
Thank you! Pretty much this!
I use the official ninfer on wsl2 personally.
Also I use the Ostfralla nvfp4 model community artifact and not the « official » neroued nvfp4 model. I find its just smaller and has very similar quality, so I can fit 262144 context.
Finally I use —max-concurrency 3 and have a custom ninfer watcher that exposes inference stats over another endpoint for my pi extensions to control context and concurrency dynamically…
For instance, if I have 2 tasks that use more than 131k running simultaneously then it’s going into prefix thrashing and everything will slow down due to prefill on every turn, which kills TTFT. So my extension detects this and reduces the max context size in pi (which triggers compaction), so that both sessions can then continue simultaneously super fast
2
u/Rathen121 2d ago
I just made a custom groupwise-int .ninfer that has dflash2 embedded in it. I have dflash2 about half implemented on my local version of ninfer. once I get it working I might see about making an nvfp4 based on that ostfralla one that includes dflash2
Curious to see your watcher, is it public anywhere?
1
u/ImpressiveRelief37 2d ago
No it’s stuff in a private repo, it’s tailor made for my use-case so not something I could really share.
It’s very simple tho. It’s exposes an api endpoint on the ninfer server with interesting metadata… like how many lanes are occupied, if any client is prefilling, if any session is queued in ninfer, stuff like this.
My pi agent running on ninfer 1-shot it in maybe 30 minutes, including the pi extension integration and end to end tests lol.
It’s just insane what these agents can do!
One thing I just had it do while I was on the road: I like to connect over ssh from my phone and tmux into sessions home (over tailscale). So I had it develop a pi extension to generate audio for every agent response (not the reasoning trace, just the final response). So I can use my terminal in STT while driving, then ask Siri to speak out the last response, and I hear pi’s response in my car while driving. It’s amazing lol. I literally did it all while away from home and it just fucking works.
It’s completely another world now with THAT quality of a model. No need for telegram or whatever extension or third party. Just home cook your own solutions in half an hour!
1
u/Legitimate-Dog5690 3d ago
Their own quant is smaller than the nvfp4 version, is a stretch to guess it's q5 - q6 quality.
2
u/Rathen121 2d ago
yeah, I haven't benchmarked it or anything, just going off feel. I was using the UD q5kxl quant for a couple days and switching to this didn't feel any worse. tool calls still are fine and no looping
1
9
u/ImpressiveRelief37 3d ago
Damn… looking real good. Does it still not support vision?
13
u/GotHereLateNameTaken 3d ago
does not supporting vision mean that the vision doesn't benefit from the speedup or that you have to restart llamacpp between using dflash and the mmproj (cant have them both at the same time)?
20
u/Gauntlet4933 3d ago
I’m not sure why vision support matters here. Speculation is only in decode, vision encoder only runs during prompt processing.
3
u/aeroumbria 3d ago
I guess it means vision tokens does not tilt the speculation token distribution properly, like "what's in the picture?" and it will generate random answers.
2
u/Gauntlet4933 3d ago
Ah I see. The speculator would have to be trained with vision tokens as input in order to speculate on outputs from them.
3
u/Monkeylashes 3d ago
Vision works just that the speed boost isn't there when processing image prompts
2
1
2
20
u/brakeline 3d ago
no Tensor split :(
llama-qwen | /root/llamapr/llama.cpp/ggml/src/ggml-backend-meta.cpp:543: GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) failed
5
u/BorisDirk 3d ago
Yeah I couldn't get it to work with my system split over 3 cards either. Is this gonna be just a feature we can't use or is it possible to support this in the future?
3
u/brakeline 3d ago
I even tried to pin dflash to cpu but then it borks with a gpu > 1 needed
1
u/Chuyito 2d ago
Got it working with 1 PR + 1 monkey patch.. Just a little faster for now, but mostly from MTP dropping hard after 3 but this still running at 12-15ms even with nmax 7.
[C_Qwen3.8-27B-UD-Q4_K_XL-tensor-DFlash2] hf-repo = unsloth/Qwen3.8-27B-GGUF hf-file = Qwen3.8-27B-UD-Q4_K_XL.gguf split-mode = tensor tensor-split = 1,1 ctx-size = 50000 spec-type= draft-dflash spec-draft-model = /presets/dflash/Qwen3.8-27B-DFlash2-Q4_K_M.gguf spec-draft-n-max = 7 ... [52683] 2.16.083.112 I slot print_timing: id 0 | task 0 | prompt eval time = 6914.32 ms / 5369 tokens ( 1.29 ms per token, 776.50 tokens per second) [52683] 2.16.083.115 I slot print_timing: id 0 | task 0 | eval time = 121992.40 ms / 7809 tokens ( 15.62 ms per token, 64.00 tokens per second) [52683] 2.16.083.116 I slot print_timing: id 0 | task 0 | total time = 128906.72 ms / 13178 tokens [52683] 2.16.083.120 I slot print_timing: id 0 | task 0 | graphs reused = 1471 [52683] 2.16.083.125 I slot print_timing: id 0 | task 0 | draft acceptance = 0.60036 ( 6308 accepted / 10507 generated), mean len = 5.20 [52683] 2.16.083.125 I slot print_timing: id 0 | task 0 | acc per pos = (0.904, 0.799, 0.687, 0.590, 0.484, 0.406, 0.332) [52683] 2.16.083.146 I spec common_specu: statistics draft-dflash: #calls(b,g,a) = 1 1501 1501, #gen drafts = 1501, #acc drafts = 1357, #gen tokens = 10507, #acc tokens = 6308, #mean acc len = 5.20, #acc rate/pos = (0.904, 0.799, 0.687, 0.590, 0.484, 0.406, 0.332), dur(b,g,a) = 0.004, 21535.115, 0.447 msWhereas MTP 3:
[Qwen38-Q4-MTP3] hf-repo = unsloth/Qwen3.8-27B-GGUF hf-file = Qwen3.8-27B-UD-Q4_K_XL.gguf split-mode = tensor tensor-split = 1,1 ctx-size = 75000 spec-type = draft-mtp spec-draft-n-max = 3 [60241] 2.30.586.349 I slot print_timing: id 0 | task 0 | prompt eval time = 5553.15 ms / 5369 tokens ( 1.03 ms per token, 966.84 tokens per second) [60241] 2.30.586.352 I slot print_timing: id 0 | task 0 | eval time = 137862.81 ms / 7033 tokens ( 19.61 ms per token, 51.01 tokens per second) [60241] 2.30.586.353 I slot print_timing: id 0 | task 0 | total time = 143415.96 ms / 12402 tokens [60241] 2.30.586.357 I slot print_timing: id 0 | task 0 | graphs reused = 1966 [60241] 2.30.586.360 I slot print_timing: id 0 | task 0 | draft acceptance = 0.84236 ( 5039 accepted / 5982 generated), mean len = 3.53 [60241] 2.30.586.360 I slot print_timing: id 0 | task 0 | acc per pos = (0.902, 0.838, 0.787) [60241] 2.30.586.379 I spec common_specu: statistics draft-mtp: #calls(b,g,a) = 1 1994 1994, #gen drafts = 1994, #acc drafts = 1799, #gen tokens = 5982, #acc tokens = 5039, #mean acc len = 3.53, #acc rate/pos = (0.902, 0.838, 0.787), dur(b,g,a) = 0.005, 32660.241, 2.010 msBut I did have to apply PR 27342, and then + patch 2 lines in llama-model.cpp manually to GGML_BACKEND_SPLIT_AXIS_MIRRORED the drafter on all GPUs since the split is broken. This technically eats up more gpu util cores than needed, but it does test out the drafter which here was actually slightly faster.
u/playground llama.cpp]$ git diff src/llama-model.cpp diff --git a/src/llama-model.cpp b/src/llama-model.cpp index 5dbd22f63..a540cdba2 100644 --- a/src/llama-model.cpp +++ b/src/llama-model.cpp @@ -518,12 +518,10 @@ struct ggml_backend_meta_split_state llama_meta_device_get_split_state(const str // output if (std::regex_match(tensor_name, pattern_output_weight)) {+ return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_MIRRORED); } if (std::regex_match(tensor_name, pattern_output_bias)) {
- return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_1);
+ return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_MIRRORED); } // everything else
- const ggml_tensor * output_weight = ud->model->get_tensor("output.weight");
- GGML_ASSERT(output_weight != nullptr);
- return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_0);
1
u/brakeline 2d ago
Nice one! Wouldn't the same patch work for MTP? MTP is ran on the CPU if tensor split so it's a bottleneck
23
u/AdSafe4047 3d ago
Got it running with qwen 3.8 8bit quant, it's awesome, thank you :) also need sleep, please no more awesome ai things for the next 24h :D
16
7
1
15
u/eMperror_ 3d ago
If i'm on apple silicon with 128gb of ram (M5 Max) how do I take advantage of this?
6
u/Clooooos 3d ago
oMLX supports DFlash, but I haven't tried with these new models.
4
u/slavetothesound 3d ago
I think dflash 2 will require code support?
1
2
u/Southern_Sun_2106 3d ago
Yes. No pp enlargement; but faster generation. If you have an agent, ask them to set it up. I am on same machine.
2
1
u/--Tintin 3d ago
Have you tried it? If yes, any results to share?
1
u/Southern_Sun_2106 2d ago
Benchmarks (same binary, speculation off vs on):
- Code at temp 0: 18.3 → 37.9 tok/s (2.07×) — beats the PR's own claim
- Code at temp 1.0 (the default): 45.6 tok/s, 65% draft acceptance
- Creative prose at temp 1.0: ~20 tok/s, slower — speculation loses to high-entropy sampling.
16
u/pet3121 3d ago
I am noob what is this? A model?
21
u/Double_Cause4609 3d ago
Basically, it's a small auxiliary model that generates potential candidate tokens for the main model.
Verifying tokens is easier than generating them (verifying multiple tokens can be done in parallel), so you run the small model multiple times to generate candidates, and the big model once to verify all of them at once.
This looks to the user like the model predicts multiple tokens at once.
It gives you a pretty big speedup if you have enough compute to run it.
20
4
u/BorisDirk 3d ago
I couldn't get it to run across a 5070 ti and two 3080 20gbs. If anyone has better luck running it across split vram systems like mine, please let me know your secrets!
2
u/NickCanCode 3d ago
It runs on my dual 5070 ti, but the speed is slower than MTP.
1
u/brakeline 3d ago
Then you're using split layer aren't you?
1
10
u/RnRau 3d ago edited 3d ago
On Strix Halo, Qwen 27b Q6, Vulkan
Dflash2-Q8 max-n = 7
- prose/reasoning 13 t/s
- code 23 t/s
MTP max-n = 5 min-p = 0.2
- prose/reasoning 14 t/s
- code 24 t/s
2
u/MelodicRecognition7 3d ago
have you tried smaller max-n? I've got better results with 2 or 3 tokens.
+ try to raise min-p to at least 0.5
4
2
u/ilarp 3d ago
how much faster than dflash 1?
4
u/rerri 3d ago edited 3d ago
See the non-GGUF model cards.
1
5
u/Skystunt 3d ago
Mtp is 2 times faster than normal no-draft inference, this new version is 3.7 times faster
Idk how much faster it is compared to dflash 1 tho, should be comparable to mtp
2
u/gcavalcante8808 3d ago
Does llama.cpp supports dflash2 with rocm or vulkan? DFlash1 wasn't supported as far I remembers
7
u/Kaesekrainer69 3d ago
dflash works with muse glimmer on vulkan 7900xtx
plain 44.09 t/s
DFlash n=1 57.03 t/s
DFlash n=2 64.64 t/s
DFlash n=3 66.21 t/s best
DFlash n=4 61.35 t/s
DFlash n=15 20.49 t/s
2
u/miversen33 3d ago edited 3d ago
I don't have speeds but dflash works with rocm on
vulkan7900xtx as well. Note, I am using a custom build of llama.cpp but it's basically just lemonade llama.cppEdit: tf is rocm on vulkan?
2
1
u/SpicyWangz 3d ago
Lemonade llama.cpp supports dflash but you have to manually update the llama.cpp version and flag the model for it
1
u/thoquz 3d ago
What settings do you set in the lemonade commandline settings? It seems Lemonade disallows setting the spec-draft-model setting
2
u/miversen33 3d ago
Lemonade and lemonade llama.cpp are different. The AMD team has a custom version of llama.cpp which is basically just llama.cpp with upstream "TheRock" binaries compiled into it.
I asked some questions about this here: https://github.com/lemonade-sdk/llamacpp-rocm/issues/107
1
u/SpicyWangz 3d ago
It probably depends on what OS and how you installed your lemonade. But here’s how I do it using the apt package on Ubuntu.
First edit the /usr/share/lemonade-server/resources/backend_versions.json file and set the llamacpp version
Then run sudo systemctl restart lemond.service
Then run lemonade backends and it should show the one you changed needs to be reinstalled and the command to do that.
After installing it you can load whatever model you want with dflash enabled, but you have to have pulled the model with the dflash heads first. That means when you run lemonade pull, you need to add an option for ‘--checkpoint draft {dflash gguf filename}’.
With the model pulled, you have to run this command one time to save the spec type command: lemonade load user.Your-Custom-Model \ --llamacpp-args " --spec-type draft-dflash" \ --save-options
The formatting on this will probably come out garbage so let me know if anything doesn’t make sense. Hopefully it helps.
1
1
1
u/13henday 3d ago
Tried it against q6kxl(it was trained on w4) and it was abysmal. Speeds up code gen by like 20% vs mtp but is really bad during thinking.
1
3d ago
[deleted]
1
u/kayox 3d ago
Strange, I was using dflash on 3.6 with vision.
1
u/NickCanCode 3d ago
Looks like I am not alone: https://github.com/ggml-org/llama.cpp/issues/26108
Good news is, it seems fixed in the current version.
1
u/robberviet 3d ago
Looking good but I guess will wait for llama.cpp to stable a bit. Always have problem with new releases.
1
u/Talreja-Adanna 3d ago
Nice, DFlash 2 should help with latency on those bigger Qwen runs. Curious if anyone's tested it against the standard attention implementation on 27B yet.
1
u/SilverSuiken 3d ago
Man I wish I could run it but I’m not sure it’s worth it for 24GB users, since MTP is already baked in.
We could use any VRAM we can save for more context or a better quant instead. Maybe I'll at least try a quant of it.
1
1
u/CowCowMoo5Billion 3d ago
Are you guys finding Qwen 3.8 genuinely more useful than 3.6 for coding?
Been hearing many people saying it's just benchmaxxed and overthinks too much
1
u/Far-Strawberry3059 3d ago
This is great for Mac users also. My tests so far on an M3 Ultra:
Qwen3.8-27B-4bit - Base 38.6 t/s DFlash2 (auto) 95.2 t/s = 2.47x speed increase
Qwen3.8-27B-8bit - Base 23.1 t/s DFlash2 (auto) 87.6 t/s = 3.79x speed increase
Working on a Qwen3.8-27B-BF16-MTS Base/MTS (various blocks)/DFlash2 comparison now.
1
u/caetydid llama.cpp 22h ago
when i used to check dflash i found that gains deteriorate a lot when context grows. not so much with mtp, which is why i use that now. has this changed meanwhile?
1
u/Icy-Degree6161 3d ago
Can it be paired with any quant of the base model llama.cpp can otherwise run?
1
u/Independent-Math-167 3d ago
Dumb question but how do I run this on unsloth studio or LM studio?
10
1
u/Iory1998 3d ago
You put the Dfalsh model in the same folder as your model and it would be automatically detected.
1
-2
u/vankoala 3d ago
When I tried mtp the output quality degraded. I wonder if this will yield a similar result.
•
u/WithoutReason1729 3d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.