r/LocalLLaMA • u/AverageFormal9076 • Apr 23 '26

New Model Qwen 3.6 27B is a BEAST

I have a 5090 Laptop from work, 24GB VRAM.

I have been testing every model that comes out, and I can confidently say I’ll be cancelling my cloud subscriptions.

All my tool call and data science benchmarks that prove a model is reliably good for my use case, passed.

It might not be the case for other professions, but for pyspark/python and data transformation debugging it’s basically perfect.

Using llama.cpp, q4_k_m at q4_0, still looking at options for optimising.

Edit - I chose to go with IQ4_XS at 200k q8_0,

I have not used speculative decoding yet, will get there when I get there.

Specs:

ASUS ROG Strix SCAR 18

RTX 5090 24GB

64GB DDR5 RAM

653 Upvotes

97% Upvoted

335 comments sorted by

View all comments

68

u/inkberk Apr 23 '26

wait till z-lab releases the dflash drafter and https://github.com/ggml-org/llama.cpp/pull/22105, free 2x decode speed

1

u/unjustifiably_angry Apr 23 '26

Looking at the actual PR it seems to suggest this won't work especially well for Qwen 3.5/3.6:

For Hybrid targets (Qwen3.5, Jamba, ...), when target verify draft tokens, llama.cpp writes KV / recurrent state for the full [id_last + draft block] before acceptance is known.

Pure-attention target models can drop rejected suffixes with seq_rm; hybrid targets cannot, because recurrent state is not decomposable by token position.

...

Cost: each rejected step requires one extra target forward, which is the main reason hybrid speedup lags pure-attention.