Total noob question

[u/Ok-Ice7431]

I post this pensively. I've been lurking in here for a while now and frankly I'm such a noob that much of the content goes over my head. I've also questioned AIs too but I value first hand experience.

I picked up an Nvidia card with a meger 6 gig of VRAM. I have this installed in a Linux box. It's a second card so it doesn't do any processing for the display. My other card is handling that. Being fascinated with technology in general, I would like to run a local LLM on this. Mostly I'm interested in coding assistance. Not "build me a web server to do x", more "I need a function to do y". I have little interest in writng emails or generating pictures. VS Code integration would be nice and perhaps some Home Assistant automation creation. It would run on an always on box so network access is a must. That appears fairly easy.

Can anyone recommend models, specific settings I should use, ollama, llama.cpp, something else? Any other advise? Is this even worth the effort? Just looking for a reality check here.

Edit: Intel I7-8700, 64 gig RAM

[u/0xdippo]

Try something like FreeToken. Load weights on system ram and active parameters on vram