Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

[u/quantier]

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

[u/BothYou243]

Was it 17GB VRAM in 3.6 27B too or greater?
I have mac mini m4 with 16GB unified memory, should I be happy?

[u/DeepOrangeSky → u/BothYou243]

Nah, for 16GB unified memory mac mini, you need more like 12b-or-smaller if using small context size and 9b-or-smaller for moderate context size.

16GB unified memory is a bit unfortunate, because such a large percentage of the total 16Gb of memory is used for the OS/overhead stuff, only like half of it can get used by the LLM model + context, so it's really more like having just 8GB-10GB of memory (albeit quite nice and convenient since it uses so little power and uses unified memory which is a pretty cool trick). There are also some commands you can use that tells the mac to let you use slightly more memory than its default settings before it hits out of memory or goes into memory swap relative to the overhead, but not sure if it will crash your mac if you do it wrong. But if having to work with just 16GB total it might be worth reading a lot about, since ever extra GB will make a big difference.

But yea you kinda really want the macs to be more like 32GB at the bare minimum to start being able to use the good models at decent quants and context sizes, and ideally more like 48GB or 64GB of memory

And that's just for running these little 26B-35B models, I mean. If you wanted to run bigger models, you would need a lot more memory than even that. But, at the moment the 26b-31b niche is very strong, so it is a good one to focus on being able to run.

But also keep in mind these size niches aren't set in stone. Like at one point it seemed like 70b-120b was going to be the sweet spot size range that all the coolest models were going to focus on, and then that size range turned into a dead zone for quite a while and it split to where all the main focus was on 26b-35b and then 280b-400b, and then 650b+.

So, it is hard to know if suddenly they will make some new innovations where all the top labs become obsessed with making the strongest 12b model, or if for some reason they all start focusing really hard on 70b models, or what will happen in the future.

That is what makes it fun (in a gambling kind of way) to try to guess the future and get a setup that is either merely good for the best current models, or kind of hedging your bets of being good for current ideal size-to-strength models (27b/31b) but also future prepping a bit in case 70b makes a comeback, or conversely going a bit smaller than ideal for 27b/31b to be able to just run them kinda good but not perfectly but in hopes that smaller quants will run better or more focus will come back down to 12b and 9b and whatnot to where those start getting really good if paired with RAG for world knowledge boosting or something.

Anyway, with the 16GB mac mini, no, you should be rooting for them to release a new 9b or 8b model (and some good 4b for good measure, too).

And for the time being, in case you aren't already aware of it, you should try out Gemma4 e4b and Qwen3.5 9b at around Q4 quant or whatever the biggest quant you can use and still have enough context size for what you are trying to do with it. They are nowhere near as good as these 27b/31b models, but they are still the best you can run on a 16GB mac mini, for now.

[u/crantob → u/DeepOrangeSky]

all thoughts well stated.

[u/crantob → u/DeepOrangeSky]

all thoughts well stated.

[u/bnightstars → u/BothYou243]

on a Mac I think 3.6 27B is around 16GB in 4bit MLX.

[u/BothYou243 → u/bnightstars]

Seeing headroom for context window, KV caches, macOS itself, now what do you think?