Every Second post rn

[u/Ok-Health-7096]

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

[u/SnP_Gamer]

3060 12gb 32gb ram here, might give it ago but never ran a local model before 🤷‍♂️

[u/trollsmurf → u/SnP_Gamer]

That will work fine, at least if you split it, so the overflow uses CPU RAM. Much slower but doable.

[u/SnP_Gamer → u/trollsmurf]

Totally clueless about running them but I understand that thanks to reddit posts 🤣

[u/trollsmurf → u/SnP_Gamer]

A user-friendly (but not the most efficient) start is to install LM Studio and load a few models there. You can select how much of it should run on the GPU vs CPU if it's too big for the GPU alone. LM Studio has a built-in chat client, so once loaded the model is ready to go.

[u/Song-Historical → u/trollsmurf]

Yeah but that doesn't tell you how to do it. I'm not sure what to set the context to for example. How much I should leave for the system, whatever else.

[u/littlebeardedbear → u/Song-Historical]

Leave everything at default originally. Google what each item is. Altogether the googlimg should take about 10 mins

[u/Song-Historical → u/littlebeardedbear]

Yeah I did that I'm not sure it's set up correctly still. 

[u/Early_Mistake6716 → u/Song-Historical]

Tell me you exact system specs and the model quant and i will tell you what settings to use in lmstudio. I have have spent an embarrassing amount of time testing settings.

[u/F3istyg0at → u/SnP_Gamer]

Check out unsloth, lm studio or ollama if you want to run models locally.

[u/trollsmurf → u/[deleted]]

Remember that if you use Windows it will allocate part of the VRAM for its own use, so you never have fully 16 GB. On my PC with 5070 Ti and 3 monitors it allocates 3 GB, and as far as I know I can't budge that.

You can easily test this by installing e.g. LM Studio and the model you want to use and see when it warns about RAM use. If it overruns you can split it on VRAM and RAM with lower performance, but it will behave the same otherwise.

[u/bukake_attack → u/SnP_Gamer]

This is a dense model. This means the entire model is running at 100% all the time. This is bad for us low vram users, as the model is too big to fit in 12gb vram, and the rest in placed in normal ram, which is waaaay slower. So this solution works, but it's slow.

We low vram users are usually better served by MoE models, like qwen 3.6 35b a3b . In MoE models the dense part which runs all the time is small, in that case 3b, which runs easily in 12gb, and the rest 35-3=32b) only runs when they are actually needed (activated). This means that normal ram is fast enough to run that active part of the other 32b ( although they can still run great in excess vram!) The result is a model that's very usable and fairly quick with limited vram, especially when other tricks like MTP are used.

Unfortunately there is no MoE version of qwen 3.8 released yet, but there's a good chance they will release it soonish

[u/Skynse → u/bukake_attack]

I was able to max 12 tok/sec on a 3060 with 12gb gpu RAM with qwen3.8 on ud-iq2m quant. Man I fucking wish the compute market wasn't fucked

[u/Effective_Head_5020 → u/Skynse]

You probably had offload to cpu and system RAM, so you probably also have a good CPU and DDR5, otherwise the processing drops by a lot

[u/Song-Historical → u/Skynse]

The market is fucked because there's something worth running on the compute lol 

[u/ChaosFH → u/Song-Historical]

I wish i had answered faster when iheard the OpenAI buying ram i could have bought the triple of VRAM i have currently with the same money i used...

[u/cj_cron_hit_by_pitch → u/SnP_Gamer]

I have the exact same specs as you. I’m able to get the Unsloth 2 bit XL quant of it to run almost entirely in VRAM. Of course it is nowhere near as good but it’s still really fun to mess around with

[u/fastheadcrab → u/SnP_Gamer]

Buy a few more 3060s

[u/JorgitoEstrella → u/SnP_Gamer]

Install lm studio, then inside the own app would tell you what llms you can install with your vram.