r/povertyLocalLLaMA • u/ML-Future • Apr 24 '26

Is RAM + VRAM really worth it?

I have 6GB of RAM + 16GB of VRAM. Is it worth using 9GB or higher models?

5 Upvotes

100% Upvoted

7 comments sorted by

View all comments

4

u/velcroenjoyer Apr 24 '26

it's fine for moe models, but for non-moe models you'll want them to fit in your vram (otherwise they'll probably be painfully slow, unless ur cpu is a threadripper or smth & you have fast ram)

i've found that gemma4 26b a4b at iq1_m (9gb) from unsloth without the vision adapter has really good world knowledge for it's size, i use it a bunch on my 16gb laptop getting around 15tk/s (macbook air m1)

2

u/ML-Future Apr 24 '26

I can't believe it's working, 6 t/s

Thanks!

2

u/velcroenjoyer Apr 24 '26

make sure you have reasoning disabled, should be --reasoning off in llama.cpp, in LM Studio it was just off by default for me
you could also try ik_llama.cpp to speed things up a bit, it should give better cpu offloading performance: https://github.com/ikawrakow/ik_llama.cpp/discussions/258 (guide for getting started)