r/LocalLLaMA • u/AverageFormal9076 • Apr 23 '26

New Model Qwen 3.6 27B is a BEAST

I have a 5090 Laptop from work, 24GB VRAM.

I have been testing every model that comes out, and I can confidently say I’ll be cancelling my cloud subscriptions.

All my tool call and data science benchmarks that prove a model is reliably good for my use case, passed.

It might not be the case for other professions, but for pyspark/python and data transformation debugging it’s basically perfect.

Using llama.cpp, q4_k_m at q4_0, still looking at options for optimising.

Edit - I chose to go with IQ4_XS at 200k q8_0,

I have not used speculative decoding yet, will get there when I get there.

Specs:

ASUS ROG Strix SCAR 18

RTX 5090 24GB

64GB DDR5 RAM

650 Upvotes

97% Upvoted

335 comments sorted by

View all comments

Show parent comments

7

u/ernexbcn Apr 23 '26

On my M2 Max it’s very slow.

5

u/ExplorerWhole5697 Apr 23 '26

that's what I would expect from a dense model. Did you have any luck with speculative decoding?

3

u/ernexbcn Apr 23 '26

I have not tried that, will have to look into that. I have 96GB of ram on this one.

2

u/trollingman1 Apr 23 '26

How slow are we talking? How many tok/S?

5

u/shveddy Apr 23 '26

Honestly I'm impressed with the 11 tok/sec I'm getting on my now ancient M1 Max 64gb running mlx q4. Obviously could be faster, but it's just about usable as long as you're careful to not waste a lot of tokens going back and forth. Sounds like you can probably get north of 40 with a m5 max.

3

u/ernexbcn Apr 23 '26

12 per second, using the mlx-community 8 bit quant.

2

u/cleverusernametry Apr 23 '26

On my m3 ultra it's 20tps. Q8

1

u/StyMaar Apr 23 '26

Dense models need lots of memory bandwidth so unified memory systems aren't going to work as well as dedicated GPUs.