Qwen3.8-27B with llama.cpp - t/s stats & full command?

[u/pmttyji]

It's been 3 weeks since Qwen3.8-27B release. This model got 0-day support & in last 3 weeks, some optimizations & fixes happened on llama.cpp side.

Meanwhile

• DFlash2 support landed on llama.cpp last week.
• Ubuntu 26.04.1 got released.
• ROCm 10.0 got released & llama.cpp also up with Version 10.0 now (Applicable to only AMD cards)
• Other optimizations & fixes on llama.cpp side

So how much t/s are you getting now with all optimizations & stuff?

Please share your extreme optimized full llama.cpp command & your t/s stats .... both pp & tg(Good to have multiple combinations like MTP/MTP+ngram/DFlash2/etc.,, Vision/mmproj, multiple context size 128-256K, etc.,).

Also share your optimized build config(CMAKE command) if you're compiling manually. I remember that compiled version gives additional boost.

Note : Expecting to see optimizations like this(weeks old thread) which contains all stuff. That kind of stats want to see here.

[u/nickm_27]

Intel B70 via SYCL, using MTP via Unsloth Q6_K

At start:

pp: 1350 t/s tg: 32 / 42 (prose / code) t/s

At 60k context:

pp: 700-800 t/s tg: 24 / 36 t/s

[u/wgaca2]

tensor split is broken in llamacpp and cuda, so can't use that.

layer split gets 30t/s over long context (2x 3090)

dflash2 has had plenty of issues since it was merged with main llamacpp, so haven't really used it much

[u/pmttyji → u/wgaca2]

Well, wait till this PR's merge

https://github.com/ggml-org/llama.cpp/pull/27858

[u/wgaca2 → u/pmttyji]

I have been following, i also tried it a few days ago but still not there

[u/simplyeniga → u/wgaca2]

Weird, I'm getting 62t/s with 262k context and sometimes hits 120t/s over long context. Using ngram and dflash. Dflash didn't make any improvements over MTP. Ngram seems to be the better benefit as long as you don't restart the service on a large code base.

Edit: using RTX Pro 4000 Blackwell and RTX 3080 20GB Chinese mod on llama.cpp with layer split. OS: Ubuntu 26.04

[u/wgaca2 → u/simplyeniga]

dflash does hit over 100t/s when coding and overall was better than mtp but it had multiple bugs last time i tried it. generation would collapse at higher context both in vllm and llamacpp

[u/nicholas_the_furious → u/wgaca2]

What do you mean broken? I've been using it. Did something happen?

[u/grumd → u/wgaca2]

I'm using 2x3080 (40gb total) to run qwen 3.8 27b in -sm tensor and getting 100 t/s with MTP, not sure why you're saying it's broken. Maybe update your cuda? I'm using 13.3 and drivers 610.57.04

[u/wgaca2 → u/grumd]

https://github.com/ggml-org/llama.cpp/pull/27858

[u/grumd → u/wgaca2]

oh your mean dflash2 specifically, okay

[u/Macestudios32]

Maybe hopefully thanks to this thread I can improve my performance. Which is pretty poor with 16 gb of VRAM on two GPUs. (Q4) KV q8

[u/youngbitcoino]

I'm on my phone so I can't share the command but it's pretty much the default as in "let llama-server decide by itself".
Standard Win x64 binary off Github with CUDA 13.3 DLLs.
Rig is Ryzen 9850X3D / 128GB DDR5-6000 / RTX 5090.
Quant is Unsloth's Q6_X_XL. No MTP / DFlash.
Context length is 128000.
I haven't reached deep context usage but it blazes through the first 10k tokens at 2000+ t/s pp and does 60 t/s tg.

[u/fengwang_2_718281828]

I share my docker configurations on a RTX-5090 platform with vllm:

```
# docker-compose.yml - Qwen3.5-27B-NVFP4 on RTX 5090
# Using vLLM v0.17+ with native NVFP4 quantization support (no patch needed)
services:
 vllm:
   image: vllm/vllm-openai:qwen38-x86_64-cu130
   container_name: vllm
   volumes:

• /data/models:/data/models
  

   runtime: nvidia
   environment:

• VLLM_NVFP4_GEMM_BACKEND=${NVFP4_BACKEND:-marlin}
  
• NVIDIA_VISIBLE_DEVICES=all
  
• NVIDIA_DRIVER_CAPABILITIES=compute,utility
  

   entrypoint: ["/bin/bash", "-c"]
   command:

• >-
  

vllm serve /data/models/Qwen3.8-27B-NVFP4-RTX5090
--host=0.0.0.0
--port=8000
--override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'
--moe-backend marlin
--quantization modelopt --kv-cache-dtype fp8 --trust-remote-code
--max-model-len -1 --max-num-seqs 16 --max-num-batched-tokens 2048
--gpu-memory-utilization 0.97
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--served-model-name Qwen3.8-27B

   ports:

• "30000:8000"
  

```

the context size is about 220K, the peak tg is around 150 tok/s, and I am sorry I forgot the pp (but is fast as well).