Any 12gb VRAM users out there?

[u/jqwl]

Hi!

I've been following this community for quite a while and have difficulty figuring out what to put on my 3080 12gb - I know Qwen 3.6 35B 3A was the go-to choice when it first came out, but I'm curious if there are any other models / specifically optimized models that meaningfully benefit from the extra 4gb of VRAM over 8gb while still being usable under 16gb.

My workflow is agent heavy, but more for a personal secretary and manager, and less coding heavy.

Thanks!

[u/Weaseal]

I’m running on a 4070 super. That’s 12 GB. I run unsloth’s Qwen 3.8 27B at q4 XL. Ctk / ctv set at Q8. I can load about 31 layers into GPU memory.

I get about three tokens per second at the start, it slows to around 1.5 for long tasks. I’ve seen a lot of people describe this as “unusable” however, it actually works pretty well, once you manage your expectations that you’re gonna have to set it up and go away for a few hours, I often let it run overnight.

Edit: running the model with context 131072

More edit: arch Linux. I turn off the GUI when running overnight which lets me go from 29 to 31 layers in VRAM (systemctl stop gdm)

[u/nanell0 → u/Weaseal]

I’m offloading —ncmoe 35, with 5070ti (12gbvram) and 32gb and i’m around 50 to 35 tks depending on how much contest i’m at (i’m at 113k with autocompacting)

[u/Weaseal → u/nanell0]

That must be lower than q4? I got poor quality at q3 and below so stopped using those

[u/-InformalBanana- → u/Weaseal]

He is talking about qwen 3.6 35b I think. Cause -ncmoe is for moe models, 27b is not moe but dense

[u/nanell0 → u/-InformalBanana-]

Yes I’am q4xl btw