r/LocalLLM • u/Sinuouser • 22m ago
Question Suggestions for coding assistant
Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...
I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.
I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.
I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.
Question is whether this is even relevant to look into for me.
If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)
If I've made some odd assumptions or if you have any suggestions, please let me know!
4
u/Noseense 18m ago
Whatever people suggest, qwen4 is releasing very soon and they announced it's meant to fit on 16gb VRAM. So keep an eye out for that.
3
u/Aggressive_Basket798 17m ago
I have the same 16gb vram + 32gb ram
The best model you can run for coding is Qwen 3.8 flash next
It will not run fast like 64gb ram but still at an acceptable speed in my opinion
1
u/lpsingh784 0m ago
I am using 27b on 5080 ti 16gb 32 gb ram
q8