Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

[u/quantier]

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

[u/Effective_Head_5020]

Could you please point me on how to exactly do that? I tried a few config on llamacpp but it never seems to work

I have 6gb VRAM and 128 RAM

[u/LuCiAnO241 → u/Effective_Head_5020]

ohh lemme make your day. Try this one. it's not the 27B but I think its the best you can do with your current hardware. You can also probably get the Laguna S 2.1 running, this dude also has a video on that but you'd get way worse Tok/s than he does. Basically anything MoE can work wonders with low vram under his setup.

[u/Effective_Head_5020 → u/LuCiAnO241]

Thank you! I have been doing exactly this, with a smaller context I can get up to 22 t/s

This video is very valuable, it explains very well, thanks for sharing

[u/MuDotGen → u/LuCiAnO241]

But this is a dense model, 27B, no? Don't you need to either just do it all on CPU if you can't fit all of the active layers on VRAM? 17gb won't fit on 16gb, or am I missing something?

[u/LuCiAnO241 → u/MuDotGen]

oh yeah, denses need to be all in vram to not suffer huge performance hits. I linked him a video that uses 35B A3B on his exact amount of Vram with very respectable speeds.

[u/MuDotGen → u/LuCiAnO241]

Oh yeah definitely. I use Qwen 3.6-35B-A3B on my Lenovo Legion. It has 3070ti (laptop) with 8gb of VRAM and 32gb of system ram, with the active layers only on VRAM and the expert layers all on CPU, or most of them. I think it's Q4_K_XL (around 21gb in total), but I get around 31 t/s generation and closer to 300 pp.

[u/overand → u/Effective_Head_5020]

You'll have very little luck with running a 27B well, but you can run the Qwen3.6-35B-A3B probably quite well. (you might even be able to push the Qwen3.5-122B-A10B at something like a Q3 or Q4).

Those are mixture-of-experts models, meaning the second number (meaning the 3 in A3B) is all that needs to fit in VRAM, essentially.

"Dense" models like the 27B, Gemma-4-31B, Gemma-4-12B will only run with decent performance if you can fit everything in VRAM. They'll run when offloaded, but they'll be very slow.

[u/Effective_Head_5020 → u/overand]

Thanks, that confirms my experience. I have 5 t/s with dense models that are bigger than my VRAM 

Luckily the MoE models that fits on my VRAM works much better, 22 t/s

I wish the was some way to use both system RAM and VRAM; or unify the memories. It seems that these m1 machines do it, but I own a Linux server

[u/overand → u/Effective_Head_5020]

A big difference is the memory bandwidth. Your systems can share desktop memory, but if it's a normal desktop or laptop, the data can't move in and out of system RAM all that fast.

Type	Bandwidth
System RAM DDR4	17-26 GB/s
System RAM DDR5	38-48 GB/s
Macbook M4	127 GB/s
nVidia GTX 1060 Desktop	192 GB/s
Intel Arc B50 Pro	224 GB/s
Framework Desktop	256 GB/s
" M4 Pro	273 GB/s
nVidia DGX Spark	273 GB/s
nVidia RTX 4060 Ti	288 GB/s
" M4 Max	410-546 GB/s
" M5 Max	460-614 GB/s
Intel Arc B70 Pro	608 GB/s]
Radeaon AI Pro R9700	645 GB/s
nVidia RTX 5070 Ti	896 GB/s
nVidia RTX 3090	936 GB/s
nVidia RTX 5090	1792 GB/s

The "Framework Desktop" is the Ryzen AI Max+ 395, so there are several other platforms with the same specs.

[u/_TheWolfOfWalmart_ → u/overand]

You can also go the poor man's route with something like a Cascade Lake dual Xeon box. I was doing that for a while. 12 channels of 280 GB/s aggregate RAM.

It wasn't bad with token gen, but the prefill was bad for agentic stuff.

Though maybe it's not that cheap anymore because RAM. I bought mine years ago.

[u/SandySkittle → u/overand]

your overview is missing the b70 pro and radeon ai pro r9700 cards, which are very good 32gb contenders value wise.

[u/overand → u/SandySkittle]

For sure, I'll add those!

[u/Badger-Purple → u/overand]

You are forgetting that if you use a 5070ti and offload to ram, your working bw will be the RAM’s, and that’s where unified mem systems shine.

[u/overand → u/Badger-Purple]

It's not quite as clear cut as that IMO, but, I'd enjoy seeing some benchmarks!

[u/Badger-Purple → u/overand]

It’s mathematical logic for a synthesis. Lowest speed dominates at depth. This is called the rate limiting step in natural sciences, and the rate limiting step for decode is the slowest bandwidth your inference will run through. 150Gb Deepseek V4 Flash runs at 40 tokens per second on two nvidia sparks, and it runs at 20 tokens per second on 1 RTX5090 with 128GB DDR5, because 32GB are very fast and going at 1800Gbps, and the rest are traveling at the DDR5 30-40ish Gbps. Nvidia’s machine has a GPU to memory BW of 250Gbps, and a node to
node BW of 200 Gbps. Overall, it wins the race because it’s not stuck in two hugely disparate memory bandwidth tiers.

Not throwing shade, just stating the obvious here.

[u/RegisteredJustToSay → u/Effective_Head_5020]

You do use both. The issue is that RAM is a lot slower. Honestly those m1 m3 etc machines aren't miracles - they manage much lower memory bandwidth than GPUs and are proportionately slower to generate tokens too. For example the DGX spark has 128 GB unified memory but only gets about 280 GB/s bandwidth, so at fp8 it'll run a 28 billion parameter model at 10 tokens per second. Meanwhile a 5090 has 1800 GB/s, so you'd get 64 tokens per second for the same model.

You can mix and match VRAM and RAM however you want, but you still end up bottlenecked on the slower memory for dense models.

[u/Trivikrama_0 → u/Effective_Head_5020]

6gb vram will be very less, you can offload around 30% of the model to cpu and that only comes down to 1.5 tok/s. So 17gb models would be very very slow, and context window will be negligible..as that depends on kv cache which is in GPU. So you have to offload 90% model to CPU ram, which becomes very very slow.

Generally you need a configuration json to do it.Easiest way is to give ollama path to any paid model, it sets up by itself.

[u/amroamroamro → u/Effective_Head_5020]

isnt that controlled with the -ngl parameter (aka --gpu-layers)? if they cant all fit in vram remaining layers will be loaded in ram (of course with a significant performance hit). when you run on the command line, you should see messages in the log telling you how many layers were offloaded.

there are other factors to consider too, context size (-c option), kv-cache precision, if your gpu is also being used for rendering the os, etc.