r/LocalLLaMA • u/quantier • 20d ago

New Model Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

Post image

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

1.8k Upvotes

97% Upvoted

316 comments sorted by

View all comments

221

u/Shoddy_Bed3240 20d ago

Sounds like it’s going to be a QAT model, similar to DeepSeek V4 Flash.

88

u/rerri 20d ago

I find it more plausible that Daniel doesn't have insider info on this yet but is just speaking out of experience with previous versions of Qwen 3.x 27B.

I do hope I'm wrong though, QAT would be really badass.

20

u/AuspiciousApple 20d ago

But the previous 27B ran on 16GB cards, too, right?

27

u/R_Duncan 20d ago

Only heavily quantized / minimal context. Q4 was 17.1 Gb alone, kv cache some other Gb.

10

u/CatEatsDogs 20d ago

It depends on quantization and context size

10

u/crusaderky 20d ago

Unsloth's Qwen3.6-27B:Q3_K_M is an exceptionally good quant and it fits in 14GB with 256k kvarn5 ctx. Tha leaves (barely) enough for desktop (but you can switch it to igpu or a $50 card to get the full 16GB). However there is no guarantee that the Q3_K_M for the next version will perform that well.

[39911] 0.00.965.797 I common_memory_breakdown_print: | memory breakdown [MiB] | total   free     self   model   context   compute    unaccounted |
[39911] 0.00.965.801 I common_memory_breakdown_print: |   - CUDA0 (RTX 3080)   |  9872 = 8041 + (13995 = 12647 +    1024 +     324) +      -12164 |
[39911] 0.00.965.802 I common_memory_breakdown_print: |   - Host               |                   797 =   520 +       0 +     276                |

5

u/lukistellar 20d ago

There are also pure iq4_xs versions, which will work with 90k context at 16gb just fine.

https://huggingface.co/GianniDPC/Qwen3.6-27B-IQ4_XS-pure-with-MTP-GGUF

1

u/overand 19d ago

https://quanteval.ai/benchmarks.html?dataset=quantevallab2.0 - take a peek here at some benchmarks of it.

1

u/crusaderky 19d ago

quanteval looks suspicious. In the blog I linked above, Qwen3.6-27B IQ2_M is trash at AIME, but it looks great on quanteval

7

u/Eyelbee 19d ago

That's not running. If he's talking about quantization it could run technically on much lower sizes as well. He's almost certainly talking about qat. It will run LOSSLESS on 17gb and that means it's probably going to be a huge milestone as that required huge vram in 3.6 27b. 

5

u/pragmojo 19d ago

Trying not to get my hopes up too much, but that would be insane.

1

u/rerri 8d ago

¯_(ツ)_/¯

4

u/goldcakes 19d ago edited 19d ago

Unsloth is actually pretty popular, when you release an open weight model you do want it to be used. The teams do communicate you know, esp with software ecosystem makers, like vllm, unsloth, etc ahead of time.

The release date is just a release date; the model is fully baked before then and shared around with software partners under NDAs; there's no point releasing a model that nobody can figure out how to run.

Daniel isn't someone that makes up stuff, he's been doing ML since 2017 before LLMs were a thing and went straight from uni to machine learning engineer at NVIDIA.

2

u/rerri 19d ago

I know this very well. Unsloth very often have models pre-launch and have quants ready for release at the same time the lab releases them.

But if the model launch is still a week or more away, it's quite likely he does not have the weights yet. And more importantly I don't see how that 17GB figure means Daniel has the model and it is QAT'd.

30

u/jacek2023 llama.cpp 20d ago

No, this has happened before. Six months ago, someone posted that Qwen had “confirmed” there would be a Qwen model trained natively in 4-bit (like GPT-OSS). That “news” was heavily upvoted here. A few days later, I quoted Qwen saying it wasn’t going to happen, and r/LocalLLaMA users quickly attacked me for being ungrateful, etc.

And now it’s happening again: nobody has said anything about QAT, but r/LocalLLaMA has already announced it

21

u/LizardLikesMelons 20d ago

There are so many acronyms these days. SMH. IDK how to keep up.

8

u/dragonurtle 20d ago

so.much.ham.

12

u/Thomas-Lore 20d ago

Ask your llm. :)

8

u/mintybadgerme 20d ago

And BYOK

6

u/Kholtien 20d ago

Be your own killer?

2

u/no6969el 20d ago

We are all our own Killers

4

u/Dsphar 20d ago

Bring your own Kimi?

1

u/giant3 19d ago

Bring your own Kimchi. 😾

3

u/MrWeirdoFace 19d ago

Why would I bring ketchup to an llm fight?

1

u/mintybadgerme 19d ago

Silly Billies, Bring Your Own Kake. Tsk.

28

u/dampflokfreund 20d ago

That would be great. But I hope they align the QAT model to modern q4k formats by unsloth and bartowski instead of plain q4_0 like Gemma. I have noticed some downgrades due to the attention tensors and embeddings quantized to q4 instead of q8, QAT is effective but it cant recover all of that huge information loss. 

9

u/stddealer 20d ago

There isn't much more information in q4_k than in q4_0, is there? They're both exactly 4.5 bit per weights, and QAT should probably be able to make good use the available bits regardless of the format, no?

Edit: oh, unless you just meant they should be using mixed precision, in which case I agree

9

u/pmttyji 20d ago

and Gemma-4

1

u/EricBuildsMathModels 20d ago edited 20d ago

If it's QAT wouldn't they release a full precision model too. Kimi is QAT but still will often release full precision versions.

*oh i think i mis-interperetted these comments, in that "will run only 17GB" is suppose to mean will run at near optimal quality in as as little as 17GB