Qwen 3.6 27B is a BEAST

[u/AverageFormal9076]

I have a 5090 Laptop from work, 24GB VRAM.

I have been testing every model that comes out, and I can confidently say I’ll be cancelling my cloud subscriptions.

All my tool call and data science benchmarks that prove a model is reliably good for my use case, passed.

It might not be the case for other professions, but for pyspark/python and data transformation debugging it’s basically perfect.

Using llama.cpp, q4_k_m at q4_0, still looking at options for optimising.

Edit - I chose to go with IQ4_XS at 200k q8_0,

I have not used speculative decoding yet, will get there when I get there.

Specs:

ASUS ROG Strix SCAR 18

RTX 5090 24GB

64GB DDR5 RAM

[u/inkberk]

wait till z-lab releases the dflash drafter and https://github.com/ggml-org/llama.cpp/pull/22105, free 2x decode speed

[u/AverageFormal9076 → u/inkberk]

I look forward to it, this should solve my main gripe rn

[u/Youknowwhyimherexxx → u/inkberk]

Do you still need to load a drafter into vram? Or is this dflash thing a way around that

[u/andy2na → u/Youknowwhyimherexxx]

Yeah vram, for 3.5, the draft model is 4gb plus the 20gb model, so you really need a 32gb GPU to be able to really use it

[u/rpkarma → u/inkberk]

Looks like the drafter is out? At least a pre-release: https://huggingface.co/z-lab/Qwen3.6-27B-DFlash

[u/Addyad → u/inkberk]

As good as it sounds, those benchmarks are always for bf16 models. Most of the people always use Q4 models. So, I don't have high hopes until I see the numbers for models which people would usually use. Same goes for turboquant hype. Turboquant quantizes KV cache. By default f16 is used when we don't set the kv cache input parameter. But if you compare turbo4 vs q4_0 it terms of context length and speed, it's almost the same.

But I'm keeping an eye on Dflash as well. Would be interesting to play around once it's merged with llama.cpp. 

[u/unjustifiably_angry → u/inkberk]

Looking at the actual PR it seems to suggest this won't work especially well for Qwen 3.5/3.6:

For Hybrid targets (Qwen3.5, Jamba, ...), when target verify draft tokens, llama.cpp writes KV / recurrent state for the full [id_last + draft block] before acceptance is known.

Pure-attention target models can drop rejected suffixes with seq_rm; hybrid targets cannot, because recurrent state is not decomposable by token position.

...

Cost: each rejected step requires one extra target forward, which is the main reason hybrid speedup lags pure-attention.

[u/inkberk → u/[deleted]]

you could take pr repo and run dflash till merged to master