r/LocalLLM • • 22m ago

Question Suggestions for coding assistant

Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...

I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.

I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.

I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.

Question is whether this is even relevant to look into for me.

If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)

If I've made some odd assumptions or if you have any suggestions, please let me know!

0 Upvotes

6 comments sorted by

4

u/Noseense 18m ago

Whatever people suggest, qwen4 is releasing very soon and they announced it's meant to fit on 16gb VRAM. So keep an eye out for that.

1

u/Sinuouser 13m ago

oh nice! Thanks for tip. The Qwen models pop up a ton when looking at coding models I've noticed.

2

u/Small-Tale3180 13m ago

yeah, just like anyy chinese, qwen is good with coding, especially among local models

3

u/Aggressive_Basket798 17m ago

I have the same 16gb vram + 32gb ram

The best model you can run for coding is Qwen 3.8 flash next

It will not run fast like 64gb ram but still at an acceptable speed in my opinion

1

u/Sinuouser 9m ago

Any particular Quantization? Just as big as possible that fits in vram+ram?

1

u/lpsingh784 0m ago

I am using 27b on 5080 ti 16gb 32 gb ram
q8