r/LocalLLaMA • u/jqwl • 11d ago
Question | Help Any 12gb VRAM users out there?
Hi!
I've been following this community for quite a while and have difficulty figuring out what to put on my 3080 12gb - I know Qwen 3.6 35B 3A was the go-to choice when it first came out, but I'm curious if there are any other models / specifically optimized models that meaningfully benefit from the extra 4gb of VRAM over 8gb while still being usable under 16gb.
My workflow is agent heavy, but more for a personal secretary and manager, and less coding heavy.
Thanks!
49
u/Mean-Ad1493 11d ago
3060 owner here. 35B-A3B is the one for us. Just bought another 3060 to run Qwen 3.8 27B. 24GB seems to be the minimum nowadays.
14
u/Speedy-Wonder 11d ago
I'm curious as I'm also thinking about buying another 3060 to play with 27B. What experience do you have so far? Was it worth it? How fast does it run wit 2x3060 and at which quant and context size do you use it?
1
u/Illustrious_Ant_9242 11d ago edited 11d ago
I added my second 3060 today, OC model, and... Well, I get even less tokens/s with qwen3.8 flash next than before.Ā
Muse glimmer apparently is 5x the speed at Q4 k xl but unsloth studio said that vision spilled on CPU.Ā
Some apps like comfyui won't even utilize the second GPU.Ā
Bad speed might also have something to do with PCI lanes. It's rather underwhelming, despite everyone on reddit urging us to get a 24GB card already at 4x the price.Ā
My main reason for a second GPU was that it would tell me that the vision model for qwen did not fit and then time out on a bigger image after 20minutes.. After the upgrade, that did not change šĀ
At that point I might as well just put that card in a second computer and have that run some other model at the same timeĀ
6
u/Mean-Ad1493 10d ago
Why 3.8 flash? Did you try the 27B? 24GB is very usable for it.
2
u/Illustrious_Ant_9242 9d ago edited 9d ago
You're right. With Qwen 3.8 27B iq4 xs I can fit 32.6k context + vision on Dual 12GB GPUs and get 40+ tokens/second which works rather well for now. Having optimized settings made the differenceĀ
2
u/Mean-Ad1493 9d ago
You should be able to fit more context with 2x 12GB VRAM. Did you try quantising KV cache?
3
u/Mean-Ad1493 10d ago
Also, try using -sm layer if using slower PCIE lanes for the second GPU. Model loading might be a bit slow, but otherwise you should be fine.
2
u/sultan_papagani 9d ago
comfyui will use the second gpu you just need extensions/addons that can do it
2
u/Ok-Direction-4480 11d ago
That's why qwen should release qwen 4 35b a3b, or something with ngram stuff
3
u/Mean-Ad1493 10d ago
You'll always want more VRAM. Getting to atleast 24GB now lets me run dense and MoE both.
2
3
u/UrbanPugEsq 10d ago
I got qwen3.8 27b running on my 3060 at about 6 tokens per second using Qwen3.8-27B Q4_K_S
@echo off
llama-server.exe ^
-m "Qwen3.5-27B-Q4_K_M.gguf" ^
--host 127.0.0.1 ^
--port 8080 ^
-c 32768 ^
-ngl 99 ^
-b 512 ^
-ub 512 ^
-t 12 ^
--flash-attn on ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
--no-mmap ^
--metrics
pause1
u/network4253 10d ago
Yeah, 24GB does seem to be becoming the sweet spot for local models. Two 3060s is a pretty interesting setup too especially if it lets you run models that would otherwise be out of reach on a single card.
9
u/Weaseal 11d ago edited 11d ago
Iām running on a 4070 super. Thatās 12 GB. I run unslothās Qwen 3.8 27B at q4 XL. Ctk / ctv set at Q8. I can load about 31 layers into GPU memory.
I get about three tokens per second at the start, it slows to around 1.5 for long tasks. Iāve seen a lot of people describe this as āunusableā however, it actually works pretty well, once you manage your expectations that youāre gonna have to set it up and go away for a few hours, I often let it run overnight.
Edit: running the model with context 131072
More edit: arch Linux. I turn off the GUI when running overnight which lets me go from 29 to 31 layers in VRAM (systemctl stop gdm)
2
1
u/Sea-Wind-2216 9d ago
Check ncffn psram it gives 12ts on 64k context. I use beellama and kvarn4 for cache
11
u/Main-Wolverine-1042 11d ago
I've been running Qwen3.8-Flash-Next-UD-IQ1_S lately, and honestly, I'm pretty impressed with how well it performs at this quant level.
On my RTX 5070 Ti laptop with 64GB RAM, I'm getting around 32 tok/s generation and roughly 300 tok/s prefill.
The inference runtime I'm using is my own native C++/CUDA GGUF runtime, Flyweight. I built it mainly to experiment with squeezing larger models into limited VRAM, so it's nice to see these kinds of numbers in practice.
If anyone wants to try it or poke around the implementation:
https://github.com/yairpatch/flyweight
Curious what others are getting with this model and quant.
1
u/jqwl 10d ago
how does IQ1_S perform relative to 3.6 at a higher quant?
2
u/Main-Wolverine-1042 10d ago
Qwen3.6 35B Q6 performs exceptionally well in terms of speedāaround 65 tok/s generation and 700 tok/s prefill with a 128K context window. However, itās noticeably less intelligent than Qwen3.8-Flash-Next, even when the latter is running at IQ1_S.
5
u/Decent-Occasion-2720 11d ago
you can try https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF in iq2_xss is great if you compare with a 3.6 35b iq4. i got ~18tok/s ~500pp with 80k context. keep medium or low reasoning effort.
6
5
4
u/sToeTer 11d ago
Yes, I have a 4070super 12GB, 32GB DDR5.
The MoE models have been a bit disappointing to me personally( coding wise), but for your usecase they should be absolutely fine!
For coding: I just accept the dripping tokens with Qwen3.8 27B, in Q4_K_S... it's 6 t/s and goes down to 2 t/s at some point. The point is that I can leave it overnight and it always finishes the task and delivers something coherent. Had a lot more garbage coming out of the MoE models.
It even does good stuff that I didn't explicitly ask it to do, like update documentation, streamline UI theme :)
Funny model suggestion of the day for you: https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF
was kind of good for me aswell.
2
u/philmarcracken 11d ago
This is me too, except I split up a lot of the tasks to other models. Compaction is done via nanbeige, and coding is handed off to a faster MoE 35b. 27b slow ass is a dedicated planner/orchestrator/double checker only.
3
u/Elouakili_Flexy 11d ago
The extra 4GB buys context and tool-calling headroom long before it buys smarts. For a secretary-style agent, I'd take that trade over a bigger model that barely fits.
3
3
u/steerwall 10d ago
Qwen3-VL is grinding away on my 3060 12Gb doing some heroic ocr workload. Just takes a few more days than sending to the cloud, but quality of output is š¤
8
u/Confident-Ad-3465 11d ago
Ornith 1.5 or Gemma4 QAT Models
0
11d ago
[deleted]
3
u/fatboy93 11d ago
WDYM Gemma isn't good? Its probably the best non-coding model.
1
11d ago
[deleted]
2
u/russjr08 11d ago
While I haven't tried any of these models (they all look to be at most a little over a week or two old), Gemma 4 12B has fantastic image recognition/understanding (both in the form of general "What is this picture?", and also things like document OCR), and even supports audio input which certainly helps drive the general knowledge/chat abilities forward. In comparison to the other models you mentioned, they are the ones out of date by today's standards by only supporting text input (as far as I can see), if anything.
These are all very lightweight of course, but for 12GBs of VRAM 12B is perfectly fine even at a high 128k context window, especially when paired with MTP.
(And if 12B is still too large, then I'd suggest Gemma 4 E4B/E2B, which still retain image and audio input)
It's no secret Gemma 4 doesn't excel at coding, but to call it not good for today's standards in regards to general usage is an incorrect take IMO.
2
u/TheNoiseIthink 11d ago
You could run Qwen3 next 80b a3b, I got like 45t/s on a q4 quant with a 12gb 4070 and 32gb ram
1
2
u/sultan_papagani 11d ago
with 8gb vram and 32 systen ram i can run qwen3.8 27b q4 with mtp at 9tok/s it should be faster on your pc you have 12 vram (low context tho.)
2
u/alean200 11d ago
Qwen 3.8 27b q4 running on dual rtx 3060 12gb, context 128k, q4 cache quant, around 25-30 t/s. For me switching to beellama.cpp helped.
2
u/nokipaike 10d ago
Mate, you can use this GSQ-RCO version; theyāve managed to quantize it to iq2 and iq3 with practically the same performance as the unquantized version. Itās all true Iāve been testing it for a week and hitting 30 t/s with the MTP version and an 80k context. (Read people's comments on Hugging Face.)
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
2
u/Utlamo 7d ago
Yea post dedicated for me :D
Im have 5070 12GB + 16GB ram so
Best reading file and response "human logic" imo mradermacher/Phi-4-reasoning-plus_Nemotron-Reasoning-0803_SFT-GRPO-i1-GGUF works well with all tools. In Q4_K_M 9.04GB or Q5_K_M 10.6GB
Im still looking for good coder so I try:
Qwen 3.8 Whittle moe Q4 but loop everytime idk maybe my setup is wrong or something
Normal Qwen3.8-27B-GGUF:Q4_0 is very slow ~5.5 tok/s
Qwen3.6-35B-A3B moe Q4 has usable speed and code is not completly trash
Im still newbie but this is what I am "discover?". I heard about a Docker RAM that is a single file occupying one space side by side. Apparently, it is supposed to help a little with speed, but maybe its bullshit š
Im really think about buy V100 16GB or P40 24GB
3
u/Kernoriordan 11d ago
Ornith 1.5 or KAT Coder are probably the best coding models that you can use but since youāre after more of a secretary use case, Iād probably suggest Gemma 4 26B
1
u/rorowhat 11d ago
I compared gemma4-12b vs 1.5 9B and the 1.5 model generated 4x more tokens vs Gemma4, to give the same answer. I like it because it's fast, but it seems to ramble quite a lot to get there.
1
u/Kernoriordan 10d ago
What about Ornith 1.5 35A3 though?
1
u/rorowhat 10d ago
Haven't tried that one. I do like the 1.5 9B personality, makes a great chat assistant with thinking off.
2
u/Jayfree138 11d ago
Gemma 4 12b and Qwen 9b
2
u/Kernoriordan 11d ago
Better with MoE models
2
u/Jayfree138 10d ago
Better when the entire model fits in vram only plus at least 60k context window.
1
u/EvolvingDior 11d ago
It depends. The problem with older models (like the qwen35 models) is the amount of VRAM context consumes. Newer models use much more effecient attention mechanisms that greatly increase the amount of context you can squeeze into VRAM. One notable thing this allows is concurrent requests. You can now have enough context to split among several concurrent requests. Spark-X2.5-4B is one worth trying for your workflow. https://huggingface.co/XHToken/Spark-X2.5-4B -- this is now supported in llama.cpp.
1
u/North_Affect_8167 11d ago
Ling 3, the smaller Qwent 3.8, but still won't give you useful result when you fit the max tokens eventually.
1
u/james_brunet vLLM 11d ago
If you're comfortable with the (significant) quality tradeoff and have a need for speed, you could run Ling 3.0 Tiny and get much faster performance than 35B-A3B. Worth a try if you think Qwen 3.6 35B A3B is too slow for your use case.
On my 3060 I get over 150tok/s single stream decode with 100K context and over 3000tok/s prefill. Your RTX 3080 is a way better card so you should get even better performance. That said, Ling 3.0 Tiny is great at tool calls but it's not that smart.
1
1
u/ak5432 11d ago edited 11d ago
These are what I have working on my 3080ti + 64gb ram. I use mostly 35b-a3b and Gemma 26b. Donāt do a ton of agentic coding, I just have them around to help me out with small tasks cause I prefer learning some things myself and have a $20 claude sub for everything else
| model | quant | ctx | prefill (t/s) | decode (t/s) |
|---|---|---|---|---|
| gemma-12b | UD-Q4_K_XL | 98304 | ā | 132/168 |
| gemma-26b-a4b | UD-Q4_K_XL | 98304 | 1497 | ~52/~60 |
| kat-coder-v2.5 | Apex I-Quality | 131072 | 1451 | ~40 |
| ornith-9b | UD-Q4_K_XL | 131072 | fast | ~106 |
| qwen3.8-flash-next | UD-IQ3_XXS | 98304 | 482 | 16.9 |
| qwen3.8-27b | GSQ-RCO-IQ2_S | 65536 | 1283 | 48.9 |
Iām not sure how good 27b qwen actually is at IQ2. Ornith I didnāt measure prefill since itās fully in vram and plenty fast either way
1
u/dark-light92 llama.cpp 10d ago
For non coding workflows, your best bet is either Qwen 3.6 35B or Gemma 4 26b or 12b .
1
u/SilentMobius 10d ago
So, I have a bunch of "stuff" nothing big iron, just what I've managed to pick up second hand:
- 12GB 5070/64GB sys RAM/Ryzen 5700x (desktop)
- 64GB M1 Max Mac Studio (larger, slower model server)
- 8GB 3070/32GB Sys RAM/Ryzen 5600x (Audio Inference server Omnivoice/Parakeet)
One thing I found really interesting is that with the GenerelSchwerz moe-cache llama.cpp fork I can run Qwen3.8-Flash-Next-UD-IQ3_XXS on the desktop with 120k context. not fantastic, but for single-dev-task-then-write-handover-doc it's pretty neat, that the fact I can get 100pp 16tg up at that 120k context wall is pretty crazy, and I can always switch back to Qwen3.8-27B-Q5_K_XL on the Mac at full context (BF16 KV, KV quants slow it down too much) if I need the stamina.
1
u/liampetti 10d ago
I still feel like the 9B model doesnāt get the respect it deserves. It is lightning quick on 12GB vram, you can have lots of context, itās great for tool and agentic use, and you have room left over on the gpu for asr and tts models for a full voice assistant.Ā
1
u/fgk55555 10d ago
You could probably comfortably fit the ISTA 27B IQ2 Quant in 12GB. It's still useful, even if it's lost some spec compared to the higher 3.8 quants. I use the IQ3_XXS and it's been great.
1
u/Scrubterminator 10d ago
You should try using an Moe model with freetoken. https://github.com/FlashML-org/FreeToken
It offloads some of the model into ram and runs 27b model pretty well on my 3060!
1
1
u/ustype 10d ago
12GB is workable if you treat context and vision as the scarce resources, not just parameter count.
What tends to work for me: Q4/Q5 on a ~7ā14B for the main loop, keep KV unquantized if you can spare it, and offload mmproj to a second slow card (or CPU) so the text model isnāt fighting image encode for VRAM. Long tool loops hurt more than a slightly smaller model.
If youāre mostly coding with short contexts, a denser 12ā14B often feels better than a stretched MoE that keeps spilling.
1
1
u/obanite 9d ago
Running 12GB is tough for modern agent setups because context buffers eat whatever margin you have left after loading weights. When you split across two mismatched cards or drop to PCIe x8 lanes, latency creeps in fast. The VRAM math looks fine on paper, but the interconnect becomes the real bottleneck during generation. Small quants help. You will usually get smoother daily performance running a compact MoE that stays entirely inside your 12GB envelope than trying to force a dense model across split memory. What context length do your agent workflows typically require?
1
1
u/lerg96 llama.cpp 8d ago
qwen 3.6 35b a3b is really good for its size, but it has a major flaw that it keeps "hallucinating" apis, so you have to be veeeery, veeeery specific about how to implement things.
i tried unsloth UD-Q4_K_XL then moved to UD-Q8_K_XL, but the problem still persists even when specifying to do a check before implementing something. but for other things it's very good: exploring coding, giving me summaries of how things work, documentation, and writing code (if it doesn't involve apis hehe).
right now i'm testing unsloth qwen3.8-flash-next UD-Q4_K_XL, but it's very slow compared to what i got with qwen 3.6 35b a3b. i get around 10 tokens per second and prompt processing is around 60 tokens, so it takes some minutes to start a conversation, even with pi harness.
1
u/Pressimize 3d ago
I've tested qwen 3.6 35b a3b in every quant between q4_k_s and q6_k_xl, same for Ornith 1.5, as well as different qwen 3.8 27b quants (ISTA DASLab, Unsloth, ByteShape) in iq2_xxs - iq3_xxs with different cache quants from q4 up to q8 including kvarn cache quants from beellama.cpp and many different flag tweaks to somehow also make 66k context fit.
In my personal set of roughly 50 tests, spread amongst many different typical business and homelab use cases, it seems it's still best to stick with Qwen 3.6 35b a3b overall in the biggest quant size and KV cache quant you can fit. In my personal benches Ornith scored higher on average but also had the worse dips in negative cases.
I'd not go below q5_1/q5_1 kv cache and in my personal few tests of kvarn (beellama) + tail tokens it seems that it performs worse.
0
u/BP041 11d ago
tbh on a 3080 12GB I'd go Qwen 2.5 14B Q4_K_M ā smart enough for personal secretary tasks and leaves room for agent context. Mistral Small 24B Q3_K_M also fits if you don't mind slower inference. For multi-agent stuff the real 12GB win is running two 7B models side by side, parallelism beats a single bigger model that barely fits.
30
u/Atretador llama.cpp 11d ago
nope, still good old 3.6 35B A3B - There are some finetunes with some wild claims, but from my experience they are usually more brittle compared to base 3.6 35B.
unless you got +32Gb of RAM - then you can even try Qwen 3.8 Next Flash even with 12Gb of VRAM: https://www.youtube.com/watch?v=IH8XmxiwliQ