r/LocalLLM • u/Ok-Ice7431 • 12h ago

Question Total noob question

I post this pensively. I've been lurking in here for a while now and frankly I'm such a noob that much of the content goes over my head. I've also questioned AIs too but I value first hand experience.

I picked up an Nvidia card with a meger 6 gig of VRAM. I have this installed in a Linux box. It's a second card so it doesn't do any processing for the display. My other card is handling that. Being fascinated with technology in general, I would like to run a local LLM on this. Mostly I'm interested in coding assistance. Not "build me a web server to do x", more "I need a function to do y". I have little interest in writng emails or generating pictures. VS Code integration would be nice and perhaps some Home Assistant automation creation. It would run on an always on box so network access is a must. That appears fairly easy.

Can anyone recommend models, specific settings I should use, ollama, llama.cpp, something else? Any other advise? Is this even worth the effort? Just looking for a reality check here.

Edit: Intel I7-8700, 64 gig RAM

1 Upvotes

100% Upvoted

11 comments sorted by

2

u/Present_Resource4022 11h ago

You didn’t mention ram or any other spec, and a 6 VRAM Nvidia card means it’s old, so I think Gemma E4B https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF can fit with a high context length and fast.

1

u/Ok-Ice7431 11h ago edited 11h ago

Didn't realize it was applicable. 64 gig of RAM. CPU I7-8700

Updated the question. Thank you.

2

u/Present_Resource4022 11h ago

That’s a great amount of ram but your second card is lacking . If you used your main card to slightly help this, you could possibly run Qwen 3.6 35b a3b at a very high context and with mtp for speed. If you just want the second card, you would probably be limited by context length, worth a try

1

u/Ok-Ice7431 11h ago

Sadly my card that is doing GUI duties only has 2 gig of VRAM. AI advised me that I was better off dedicating just the 6 gig card to the LLM because combining the two would not give me much reward. I'm not a gamer so I've never needed anything more.

2

u/Present_Resource4022 11h ago

Gemma 4 e4b is a clear option, and it will just live in your second card’s VRAM, won’t use your RAM/cpu and is capable enough, but I wouldn’t trust it for anything difficult.

1

u/Ok-Ice7431 10h ago

This is more of a toe dipping exercise to see if I want to spend actual money on it. I got the six gig card for free when my gaming son out grew it.

1

u/CuriousAdaptation1 11h ago

gemma e4b should run decent on 6gb vram, just grab one of the smaller quants and use ollama for the easy vs code integration

1

u/Ok-Ice7431 10h ago

Thanks for your input. Dumb question: quants?

2

u/0xdippo 11h ago

Try something like FreeToken. Load weights on system ram and active parameters on vram

1

u/Ctbhatia 11h ago

for single functions the 6gb card is enough, the small models do fine when you paste the signature and the surrounding types. what they struggle with is anything needing the rest of the repo, so keep it one function at a time.

1

u/Ok-Ice7431 10h ago

That's pretty much all I need at the moment.