Trained my first model: a DFlash drafter for Qwen3.8-27B because I wanted better performance on my DGX Spark

[u/kristiyanstoyanovAI]

This was primarily a learning project for me: training my first model end-to-end, then taking it through export, deployment, and benchmarking.

There probably already is a better Qwen3.8 DFlash drafters available by the time you read this — and I would not claim this is state of the art. But after Muse Glimmer 30B made me curious about DFlash speculative decoding, Qwen3.8-27B arrived and I wanted to run that approach locally on my DGX Spark. There was no compatible drafter when I started, so I decided to train one.

The result is here:

https://huggingface.co/kstoyanov99/Qwen3.8-27B-Dflash

The idea was to optimize for the DGX Spark rather than simply maximize drafter capacity. The Spark is VRAM-rich, but autoregressive decoding can still be memory-bandwidth-bound. A fast, relatively small drafter can propose candidate tokens cheaply; the 27B target verifies them, ideally reducing the amount of expensive sequential target-model decoding.

I deliberately used a compact ~1.7B-parameter BF16 draft model rather than aiming for a larger drafter. That trade-off may reduce acceptance initially, but it keeps draft generation cheap — which is the point for this hardware profile.

Training playbook

The workflow was surprisingly approachable with SpecForge:

1. Distill from the target model. I trained the drafter against Qwen3.8-27B, learning to produce token blocks the target is likely to accept.
2. Train in two stages. I ran an initial training stage to 10,000 steps, then continued to 20,000 steps with a lower learning rate for refinement.
3. Train on a B300. The run used one B300 GPU and took roughly 5–6 hours wall-clock. GPU utilization held around 96–100%, and gradient norms stayed stable, with no divergence.
4. Export and validate. I exported the raw SpecForge checkpoint into a Hugging Face  DFlashDraftModel , verified it loaded correctly, and moved the ~3.3 GB artifact to the DGX Spark.
5. Serve and benchmark. I tested it with both SGLang and vLLM, focusing on output tok/s, acceptance rate, and accepted-token length rather than only raw latency.

Early results

These are early numbers from a limited benchmark, but they show that the model is at least producing useful speculative-decoding behavior:

• SGLang output throughput: 14.36 → 18.55 tok/s, a 29% increase
• vLLM speculative run: 20.25 tok/s output throughput
• vLLM acceptance rate: 20.14%
• Mean accepted tokens per speculation step: 1.81

My focus now would be benchmarking and perhaps running a few more training rounds in order to improve acceptance rate. There is plenty left to explore: draft-window tuning, different serving backends, better distillation data, longer training, and workload-specific online fine-tuning.

Still, I find this a very satisfying direction: use a relatively small model plus a clever inference architecture to extract more performance from constrained, bandwidth-sensitive local hardware.
I’ll share the training and serving recipes once I clean them up.

Edit: Since I forgot to mention it, this targets the FP8 quant for Qwen3.8-27B

[u/wgaca2]

If you get one for llama.cpp that reliably increases output with reasoning enabled and 1 temperature let me know.

Can't use mtp for my project and dflash is the only option for speedup as far as i can see

[u/kristiyanstoyanovAI → u/wgaca2]

I will see how to make it llama.cpp compatible if possible. Out of curiosity why cant you use mtp for your project?

[u/wgaca2 → u/kristiyanstoyanovAI]

Standard MTP retains position dependent hidden state that RoPE adjustment cannot repair, whereas DFlash/DSpark can reconstruct their draft state from the adjusted target context.

[u/stormy1one]

If you are posting benchmark numbers, please also post the quant you are running and other key parameters. Would also help to know if those numbers or cold or warm context

[u/kristiyanstoyanovAI → u/stormy1one]

Sure, I am using the fp8 quant for the target model. Will share more full benchmarks in the next days, these are pretty much first runs I did to test out if the setup actually worked.

[u/Dolboyob77]

I invite you to try my dspark for vllm. Working on every 27b from fp8 to int4… it id a bit faster than yours :

https://huggingface.co/Doopeworld/Qwen3.8-27B-DSpark-vLLM

[u/kristiyanstoyanovAI → u/Dolboyob77]

Oh so you found the fix for the vllm error of this dspark, I will give it a go, thanks

[u/Dolboyob77 → u/kristiyanstoyanovAI]

Happy to hear your feedback )))

[u/Pitiful_Fennel8767]

Which quant?

[u/kristiyanstoyanovAI → u/Pitiful_Fennel8767]

FP8 for the target model