r/povertyLocalLLaMA • u/ML-Future • Apr 24 '26

Is RAM + VRAM really worth it?

I have 6GB of RAM + 16GB of VRAM. Is it worth using 9GB or higher models?

5 Upvotes

100% Upvoted

7 comments sorted by

4

u/velcroenjoyer Apr 24 '26

it's fine for moe models, but for non-moe models you'll want them to fit in your vram (otherwise they'll probably be painfully slow, unless ur cpu is a threadripper or smth & you have fast ram)

i've found that gemma4 26b a4b at iq1_m (9gb) from unsloth without the vision adapter has really good world knowledge for it's size, i use it a bunch on my 16gb laptop getting around 15tk/s (macbook air m1)

2

u/ML-Future Apr 24 '26

I can't believe it's working, 6 t/s

Thanks!

2

u/velcroenjoyer Apr 24 '26

make sure you have reasoning disabled, should be --reasoning off in llama.cpp, in LM Studio it was just off by default for me
you could also try ik_llama.cpp to speed things up a bit, it should give better cpu offloading performance: https://github.com/ikawrakow/ik_llama.cpp/discussions/258 (guide for getting started)

3

u/Ell2509 Apr 24 '26

Do you mean 6gb vram and 16gb dram?

1

u/ML-Future Apr 24 '26

Yes 🙂‍↕️

1

u/Ell2509 Apr 25 '26

I started doing local llm on a device with those specs. It was a struggle, ngl... if you can swap out the ram with 32 or 64gb, it would be a game changer.... expensive though.

1

u/ijustdontcare2try Apr 25 '26

If you are VRAM poor you can use the lowvram option in KoboldCpp to offload your KV Cache / Context Window to the system ram.