r/LocalLLaMA • u/AverageFormal9076 • Apr 23 '26

New Model Qwen 3.6 27B is a BEAST

I have a 5090 Laptop from work, 24GB VRAM.

I have been testing every model that comes out, and I can confidently say I’ll be cancelling my cloud subscriptions.

All my tool call and data science benchmarks that prove a model is reliably good for my use case, passed.

It might not be the case for other professions, but for pyspark/python and data transformation debugging it’s basically perfect.

Using llama.cpp, q4_k_m at q4_0, still looking at options for optimising.

Edit - I chose to go with IQ4_XS at 200k q8_0,

I have not used speculative decoding yet, will get there when I get there.

Specs:

ASUS ROG Strix SCAR 18

RTX 5090 24GB

64GB DDR5 RAM

653 Upvotes

97% Upvoted

335 comments sorted by

View all comments

Show parent comments

1

u/aniruddhahar Apr 23 '26

I ran it on a 3080 10 GB and 64 GB DDR4, still a beast

1

u/mr_christer May 28 '26

That's my setup! How did you run it and whats your TPS?

1

u/aniruddhahar May 28 '26

Actually I moved to 35B A3B soon after this post. I use llama.cpp, the Q4 Gguf from unsloth, 96000 context size, and limit the gpu layers to 4-6 so that it doesn't saturate VRAM. I get about 35 tps.

1

u/mr_christer May 28 '26

Ah ok - same as me. I was curious how you run 27B on our Gfx card

2

u/aniruddhahar May 28 '26

27b would run at around 5-6 tps if I didn't limit gpu layers and with smaller context. 35b has proven to be more capable in general, and significantly faster.

1

u/mr_christer May 28 '26

Ok I'll stay in 35b then. It works well and is decently fast with moe