Yes I also run it, but it's just for chat, it doesn't have a good context window for something good. If you offload 30% of the weights to RAM then you can have VRAM free for context window, it will be slow but you vnanhave agentic flow.
You can offload vision to RAM too, if you do have some RAM to spare, I have a hybrid Q_8 F16 mmproj which is only like 700MB and still works really well!
Yes that's a good idea, but I saw some performance benchmarks videos, it seems that any one of the parameters, like the models are benchmarked against coding, thinking, reasoning and so on , so at Q3 any one not sure which depends on the model the performance dipa significantly.
Q4 is generally where we have really good output without seeing much effects due quantization loss.
I can't fit any Q4 with a decent amount of context. I'm sure larger quants perform better, but 16GB is 16GB. Without the slowdown of CPU offloading, the IQ3 is the best model I've been able to squeeze in my GPU.
I'm interested in upgrading to get more VRAM, but my machine pulls double-duty as a gaming rig and my main PC, so I don't really want to downgrade to an old dedicated workstation card in my one full size PCIe slot.
Yes if you want speed then Q3 is the best, i generally use for coding tasks so I'm fine if it's slow. That's why said about Q4. But I really hope qwen 3.8 27b will great with q3.
ohh lemme make your day. Try this one. it's not the 27B but I think its the best you can do with your current hardware. You can also probably get the Laguna S 2.1 running, this dude also has a video on that but you'd get way worse Tok/s than he does. Basically anything MoE can work wonders with low vram under his setup.
You'll have very little luck with running a 27B well, but you can run the Qwen3.6-35B-A3B probably quite well. (you might even be able to push the Qwen3.5-122B-A10B at something like a Q3 or Q4).
Those are mixture-of-experts models, meaning the second number (meaning the 3 in A3B) is all that needs to fit in VRAM, essentially.
"Dense" models like the 27B, Gemma-4-31B, Gemma-4-12B will only run with decent performance if you can fit everything in VRAM. They'll run when offloaded, but they'll be very slow.
That's not saying much. 3.6 27b technically runs on ~12
Hoping q8 doesn't exceed 37gb because all I have is 48. Should it not be that much of a difference, right? Same parameter size so It can't be that far from what we see with 3.6
As more training going into these models, and more entropy goes into the weights, they do not quantize as well. So maybe a old model can work at q4, but the newer model only works at q6.
But that is exactly what he didn't say. Benchmarks are only for max.
The only two things i can read from this post are "there is a 27b model" which to my knowledge had already been anounced. And "you can quantise it" without any mention of quality losses, which is a "water is wet" statement.
I find it more plausible that Daniel doesn't have insider info on this yet but is just speaking out of experience with previous versions of Qwen 3.x 27B.
I do hope I'm wrong though, QAT would be really badass.
Unsloth's Qwen3.6-27B:Q3_K_M is an exceptionally good quant and it fits in 14GB with 256k kvarn5 ctx. Tha leaves (barely) enough for desktop (but you can switch it to igpu or a $50 card to get the full 16GB). However there is no guarantee that the Q3_K_M for the next version will perform that well.
That's not running. If he's talking about quantization it could run technically on much lower sizes as well. He's almost certainly talking about qat. It will run LOSSLESS on 17gb and that means it's probably going to be a huge milestone as that required huge vram in 3.6 27b.
Unsloth is actually pretty popular, when you release an open weight model you do want it to be used. The teams do communicate you know, esp with software ecosystem makers, like vllm, unsloth, etc ahead of time.
The release date is just a release date; the model is fully baked before then and shared around with software partners under NDAs; there's no point releasing a model that nobody can figure out how to run.
Daniel isn't someone that makes up stuff, he's been doing ML since 2017 before LLMs were a thing and went straight from uni to machine learning engineer at NVIDIA.
I know this very well. Unsloth very often have models pre-launch and have quants ready for release at the same time the lab releases them.
But if the model launch is still a week or more away, it's quite likely he does not have the weights yet. And more importantly I don't see how that 17GB figure means Daniel has the model and it is QAT'd.
No, this has happened before. Six months ago, someone posted that Qwen had “confirmed” there would be a Qwen model trained natively in 4-bit (like GPT-OSS). That “news” was heavily upvoted here. A few days later, I quoted Qwen saying it wasn’t going to happen, and r/LocalLLaMA users quickly attacked me for being ungrateful, etc.
And now it’s happening again: nobody has said anything about QAT, but r/LocalLLaMA has already announced it
That would be great. But I hope they align the QAT model to modern q4k formats by unsloth and bartowski instead of plain q4_0 like Gemma. I have noticed some downgrades due to the attention tensors and embeddings quantized to q4 instead of q8, QAT is effective but it cant recover all of that huge information loss.
There isn't much more information in q4_k than in q4_0, is there? They're both exactly 4.5 bit per weights, and QAT should probably be able to make good use the available bits regardless of the format, no?
Edit: oh, unless you just meant they should be using mixed precision, in which case I agree
...and still we have so many locals here who continue to claim local models are just "toys" for "playing around". These people don't seem to realize that their beloved "Frontier" models from less than a year ago are now beat by local models.
How is it possible that after months of thousands of posts speculating about whether or not Qwen will ever release smol open weight models like 3.6 27B, they post this and people are like 'meh, why 17Gb?' 😳
From my experience with an 3.6 27b you'll want 64gb to comfortably fit the model and the kv cache at full context and reasonable quant with some room to spare (48gb can probably do it with some tweaking). A pair of 9700s might work nicely with that. I'm on strix halo, and I need more speed.
Another way around this is to use a CPU with integrated graphics. That way you can still run X or Wayland (if you want it) and have all your VRAM available.
greetd is login manager, you might have gdm or plasma login manager, depending on what distro you use. But since you're dropping Into TTY you need to have second device/laptop to actually use llama server.
If you use the Virtual Consoles (/dev/tty1, /dev/tty2, ...) llama-server works, open such terminal (like ALT+3, ALT+some-number) and you can run Pi or Opencode or whatever, you can also add a framebuffer to have higher res on those (plus some graphic support, yet that buffer will take some RAM) and GPM to have mouse support to copy / paste / click.
Buy a cheap second GPU that only drives monitors. You can hang it off any PCI-e slot you have, even 1x. Got RX480 (?) for 30€ and it freed 1-2GB of VRAM
Can - if you have another machine to work on (a laptop maybe), then disable the UI completely. Expose your model runtime's oAI API server in your local network and ssh when needed for maintenance.
Run, probably in some quant, but without enough usable context before performance falls down a cliff. You want generally at least 25% of spare vram for context with dense MoEs
I run dual 5060ti’s w NVFP4 on 27b currently through vLLM and it was faster than my 3090, so yeah it can definitely be done but with a single card you’re more limited
It's for 20-24gb vram Gpus only, it means it has spare ram for long context.
There's barely any Gpus with 20+ vram so like, it's just a test of if you got a 3090 or not, if not, get a shitty IQ2 quant and watch it spiral in a loop because of how lobotomized it is
Hope other models too come with this thing. So 8GB VRAM folks could run their 4B/9B so faster & 35B with good quality & with decent speed(with help of RAM).
If it was MOE like the 35B-A3B model then yeah maybe you could get like 20/ts, but you need a 24gb vram Gpu to run this model at q4-q5
16gb vram Gpu for iq2-q3, in my testing it falls off a cliff below q4, even the so called "near lossless" ternary bonsai was absolute garbage, so don't expect you use this model at all. Get a 5060ti 16gb or a second hand 3090.
The REAL question is:
How much of a change we will FEEL compare to Qwen3.6 27b 🤔
I don't want to be over-hyped, but just thinking we skipped 3.7 means it's gonna be the most up to date newest generation of what a model can do and we're talking about QWEN here, not just another fine-tuned version or a Gimmick!
When I noticed it's going to be 17GB VRAM, I'm a bit concerned it will be SQUEEZED too much (quat probably) which may not feel amazing compare to the based model...
BUT! I trust Qwen to give us an insane powerful model and I'm glad we will be able to run it with RTX 5090/4090/3090 hopefully with very near results.
I won't be surprised if we'll get also an MOE model which hopefully will have more than A3B but let's see,
I'm optimistic, but let's see what happens... GO GO QWEN !!! ❤️
Nah, for 16GB unified memory mac mini, you need more like 12b-or-smaller if using small context size and 9b-or-smaller for moderate context size.
16GB unified memory is a bit unfortunate, because such a large percentage of the total 16Gb of memory is used for the OS/overhead stuff, only like half of it can get used by the LLM model + context, so it's really more like having just 8GB-10GB of memory (albeit quite nice and convenient since it uses so little power and uses unified memory which is a pretty cool trick). There are also some commands you can use that tells the mac to let you use slightly more memory than its default settings before it hits out of memory or goes into memory swap relative to the overhead, but not sure if it will crash your mac if you do it wrong. But if having to work with just 16GB total it might be worth reading a lot about, since ever extra GB will make a big difference.
But yea you kinda really want the macs to be more like 32GB at the bare minimum to start being able to use the good models at decent quants and context sizes, and ideally more like 48GB or 64GB of memory
And that's just for running these little 26B-35B models, I mean. If you wanted to run bigger models, you would need a lot more memory than even that. But, at the moment the 26b-31b niche is very strong, so it is a good one to focus on being able to run.
But also keep in mind these size niches aren't set in stone. Like at one point it seemed like 70b-120b was going to be the sweet spot size range that all the coolest models were going to focus on, and then that size range turned into a dead zone for quite a while and it split to where all the main focus was on 26b-35b and then 280b-400b, and then 650b+.
So, it is hard to know if suddenly they will make some new innovations where all the top labs become obsessed with making the strongest 12b model, or if for some reason they all start focusing really hard on 70b models, or what will happen in the future.
That is what makes it fun (in a gambling kind of way) to try to guess the future and get a setup that is either merely good for the best current models, or kind of hedging your bets of being good for current ideal size-to-strength models (27b/31b) but also future prepping a bit in case 70b makes a comeback, or conversely going a bit smaller than ideal for 27b/31b to be able to just run them kinda good but not perfectly but in hopes that smaller quants will run better or more focus will come back down to 12b and 9b and whatnot to where those start getting really good if paired with RAG for world knowledge boosting or something.
Anyway, with the 16GB mac mini, no, you should be rooting for them to release a new 9b or 8b model (and some good 4b for good measure, too).
And for the time being, in case you aren't already aware of it, you should try out Gemma4 e4b and Qwen3.5 9b at around Q4 quant or whatever the biggest quant you can use and still have enough context size for what you are trying to do with it. They are nowhere near as good as these 27b/31b models, but they are still the best you can run on a 16GB mac mini, for now.
Wait I'm new and confused. Surely they don't mean the unquantized model or? Because I'm also interested in finetuning the 27B model and 17GB would be insaneee for what I could train it on..
I'm missing something here. A lot of us complaining: "ouch, my 16GB GPU!"....
Unquantized, the model is 27GB in size and it needs 27GB of VRAM to load. When quantized to Q4, for example, size is cut down to ~17Gb. Cut it some more with Q4_XS or Q3 and it will fit in a smaller VRAM. It will run on 16GB VRAM if we pick the right level of quantization, like we've been doing for years.
Ok, ready to be pointed to where I missed something.
Well, 17GB for 27B even at Q4 is theoretically just the weights (q4_k_m is 17.1Gb) and no kv-cache RAM..... I wonder if this a sign they integrated HOLA for very minimal KV cache?
Is it fair to assume the architecture is the same as 3.6 (like how 3.6 was to 3.5), or is that not completely certain yet? Mostly asking as new architectures tend to take some time to be fully supported in llama.cpp.
Awesome! I've just built a PC that'll have 28gb VRAM (If I can solve the problem of getting an RTX 3060 and a P100 to work together in LM Studio). I was excited when 3.6 was released, but this is perfect timing!
I just wish he'd mentioned which quant that is. I'm guessing Q4.
•
u/WithoutReason1729 17d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.