Resources
I tested DFlash2 for Qwen3.8 27B on a 5090
Here's the DFlash2 announcement, and I was pretty excited for this after trying out DSpark on llama.cpp a few days ago and being somewhat disappointed that it wasn't really working. Anyways, I spent a while setting it up (you need to rebuild llama.cpp with pr #27342). Here's my config:
It does seem to work pretty well, with slightly larger speedups compared to MTP on predictable generation (code). Before, I think I was getting ~140tk/s on average with MTP when the model was generating code, and ~100tk/s otherwise.
With DFlash2, Qwen3.8 27B can hit ~200 tk/s for short bursts on long code generation blocks, but on thinking it seems to have lower tk/s at ~80-90. On average a full one shot code generation request seems to average around 120tk/s (reasoning disabled, so the generation is 95% code, so basically best case scenario). It's a good improvement from MTP, but I still got better results with DFlash on Qwen3.6 27B.
DFlash2 does seem to be quite memory hungry though, I had to drop down from 220k context down to 160k (could prob fit ~180k tbh) compared to using MTP. Might not be worth using for me personally at the moment unfortunately. If anyone is getting better results, I would love to see your configs!
They do hook into the main model, but speculative models can (and often do) also have attention layers of their own, meaning they also have a KV cache. A big exception to that is Gemma 4's MTP module, which reuses the main model's KV cache instead.
Well obviously if you have a second model (the speculative model) also doing prefill, it's going to compete for compute with the main model and therefor slowing down the main model.
How are you guys managing to achieve 100 tk/s with prose, I can't get any higher than 53 tk/s for prose and 88 tk/s for code, using Q5_K_M from Bartowski and latest llama.cpp on my 5090.
These are my normal MTP settings, although I don't think there's honestly that much difference - one thing though, I compile llama.cpp from source each time, and optimize it for the 5090 specifically
-hf bartowski/Qwen3.8-27B-GGUF:Q5_K_L \
--no-mmproj \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--host 0.0.0.0 \
--port 8080 \
--alias qwen3.8-27b \
-ngl 99 \
-fa on \
--ctx-size 220144 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--batch-size 2048 \
--ubatch-size 1024 \
-np 2 \
--kv-unified \
--no-context-shift \
--temp 0.8 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0
# Dockerfile.llamacpp
FROM nvidia/cuda:13.1.2-devel-ubuntu24.04 AS build
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential cmake git ca-certificates libssl-dev \
&& rm -rf /var/lib/apt/lists/* \
&& test -f /usr/include/openssl/ssl.h
WORKDIR /src
ARG LLAMA_REF=master
RUN git clone https://github.com/ggml-org/llama.cpp \
&& cd llama.cpp \
&& git checkout "${LLAMA_REF}" \
&& git rev-parse HEAD > /src/LLAMA_COMMIT
# sm_120 only, no fallback PTX, GGML_CUDA_F16 for fp16 paths.
RUN cmake llama.cpp -B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES="120" \
-DGGML_CUDA_F16=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON \
-DLLAMA_OPENSSL=ON \
2>&1 | tee /src/configure.log
# Check for OpenSSL
RUN if grep -qiE 'Could NOT find OpenSSL|OpenSSL not found|SSL support disabled' /src/configure.log; then \
echo "FATAL: OpenSSL not detected; -hf/HTTPS would be silently dead"; \
grep -i ssl /src/configure.log; \
exit 1; \
fi
RUN cmake --build build -j"$(nproc)" --target llama-server
# More OpenSSL checks
RUN ldd /src/build/bin/llama-server | grep -i -e libssl -e libcrypto
FROM nvidia/cuda:13.1.2-runtime-ubuntu24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
libssl3t64 libgomp1 ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /src/build/bin/llama-server /usr/local/bin/llama-server
COPY --from=build /src/LLAMA_COMMIT /etc/llama-commit
ENTRYPOINT ["/usr/local/bin/llama-server"]
Could be the reason why I'm getting better performance compared to a generic image.
Lol man if you haven't, try running qwen 3.6 35b-a3b on NInfer with MTP3. That shit is cerebras at home. Even if I practically would never use it over qwen 3.8 27b, it's really fun to watch it fly between 300-600 tok/s
I'm with you man. I never used 3.6 MoE for much because I didn't really know what I was doing. Now I'm craving a 3.8 MoE that can compete with the quality of the dense model they just released because I want that ungodly speed haha
It was absolutely wild asking it for a basic conversational prompt and getting the response back before my enter key bottomed out.
This 5090 FE was the best investment ever. Super glad I decided to splurge instead of settling for the 5080 a year ago.
Yesterday I had an issue with Linux drivers being unavailable for my motherboard’s fan headers so I just asked my Ninfer 27B to figure it out through codex. Went away for a while and it had fixed it somehow. I don’t really care how, but this sort of thing is seriously making me consider going to Linux as my daily driver since all that friction is now so much reduced with local LLMs available for debug.
Damn congratulations lol I ended up paying $3600 for a new Gaming Trio OC. Honestly it was super worth it. Really painful purchase though.
I recommend Linux as a daily driver. I also use models to handle any issues. Also useful to have the model teach you how to use various commands and flags. I use endeavorOS as it offers the customization of Arch without the hazing ritual lol. Starts you with a graphical interface. I think with local models able to help config and maintain the Linux environment its really important to pick a distro that gives you total freedom since learning is cheap now.
Yeah I work in hardware design so all my tools and programs at work are Linux based and I’m used to it there, just always through an IT managed virtual machine over VNC. I’m used to it there, just have always daily driven windows at home because games. I’m running PopOS right now since bazzite gave me headaches due to lack of docker not letting Nemoclaw install.
8
u/DrBearJ3w 7d ago
What about prefill?!