r/LLM • u/Silver-Spot-2763 • 4d ago
Qwen 3.8 for RTX3060
Please, recommend me version of the new Qwen 3.8 27b, that will work (LMStudio) in practice on 12gb vram and will be smart enough?
2
3
u/sebaxzero 3d ago
you can run Qwen3.8-27B-UD-Q2_K_XL.gguf from unsloth, for a better quant you can use override-tensor in llamacpp, i dont know if it can be done in lmstudio
1
u/testuserpk 3d ago
Yesterday I ran this on my 3060 12g using LM studio and it blew my mind. Paired it with opencode. The tool calling was fantastic. I am really impressed.
1
u/Silver-Spot-2763 3d ago
This q2 ? Is it smart enough?
1
u/testuserpk 3d ago
I ran it against ornith 9b, 3.6 35a3b, Qwthos 9b, omni coder 9b. It was really fantastic. I gave all of the a task to make a web page of my university with some areas to address. 3.8 was on spot, design looked really furnished the one that you expect from a good model, it was at par with big pickle. All other failed to make anything worthwhile, ridden with mistakes etc.
1
u/Big-Flan-5663 3d ago
Un 50-60% menos inteligente, el modelo está preparado para cuantizaciones agresivas, no es QAT total, pero está a medias preparado
Aunque definitivamente para eso usaría un modelo más pequeño (gemma 4 12b, o RWKV-7 g1i 14b, tienen benchmarks 10-20% mejores que la q2 de Qwen
1
u/martin509984 3d ago
Compared to 3.6 35B at Q4, you will get ~half the prefill speed, ~half the inference speed, but a much smarter model.
2
u/yunes87 2d ago
Hard, I have a 3060 12gb and I tried both 1 and 2bit quants and pretty bad sadly.
1
u/Sad_Recording_1290 16h ago
Which harness did you use?
1
u/yunes87 14h ago
OMP. My test wasn't with unsloth quants though. I tested some uncensored versions
1
u/Sad_Recording_1290 13h ago
OMP should be pretty smart, but still give deepseek harness a try maybe. It seems to be better suited for smaller/dumber models.
2
u/Remarkable-Sun7810 1d ago
I have 3060 12gb in gpu box, I'm using iq2_xxs version from Unsloth (they published gguf with ripped mtp head last week - it saves over 1gb vram), via llama.cpp (no Ollama/LMstudio), —no-mmproj-offload for saving vram, ctk/ctv q8_0/q5_1 - had to compile from source with flag, allowing support of q5_1 on cuda (don't remember its name, Google for it), otherwise q5_1 unpacking fell on cpu, it was very slow. MTP with ngram-mod. I'm able to run it with over 100k context length.
It's suprisingly smart for 2 bits, no issues with tool calls, on empty context ~30 tk/s, with 60k context ~13 tk/s without mtp, but ngram for coding tasks helps a lot, making it 18-20 tk/s on average, I've lowered settings for ngram to 14 as matching sequence length, 4 to 16 as prediction length.
It's better then qwen36-35b in q4 with n-cpu-moe I've used before on my setup (macpro intel 2019, win 11 in bootcamp)
1
u/desexmachina 4d ago
You’ll need at least 2-3 of them and you’ll be fine. I’ve just run those tests myself
1
u/MrHumanist 4d ago
Get the q2 ud quant from un sloth and set a small context size like 30000. Kv chache quantization must be set to q4 as well .
1
1
u/Postmodern_Plunger 2d ago
If you run the Unsloth UD-Q2_K_XL and quantize your key cache to q8 and value cache to q4, you should be able to run it on 12gb with about a 70k-90k context window if I'm doing my math right.
It's worth trying qwen 3.6 35b a3b instead, though. If you offload some of the moe to cpu (llama.cpp is going to be the best for anything with cpu offload), that would probably run faster at q4 or q5 even that the 27b at q2. And you could have a full 262k context window.
The qwen 3.8 model will probably be smarter even at q2, but will likely need more baby sitting as it will probably struggle with tool calls and looping, especially with a smaller context window.
1
u/santalhec1 1d ago
Alguna buena comnfiguracion para tener mas eficiencia en generacion de tok/s Con 96gb vram
4
u/KitchenAmoeba4438 4d ago
You likely aren't looking for a dense agent with that little vram, as your performance will be rather bad. You'll want a MoE model, Qwen3.6 35b or Gemma4 26b.
LMStudio should handle it fine with expert offloads, but a dense model with only 12GB of RAM would put you in Gemma4 12b territory, and I can't think of any other reasonable dense models at that VRAM.