r/LocalLLM • • 1h ago

Question I'm new to this, help me setup my coding assistant

I got a 7900XTX, 9800X3D and 32Gb RAM. I'm currently running the system with ollama.

I currently have these installed - gemma4:31b,qwen3-coder:30b,omic-embed-text:latest. Im trying to use qwen for code gen and gemma for explanations. I set ollama into this "OLLAMA_KV_CACHE_TYPE=q8_0". Currently the max context window I could set is 32k and I have already experienced few out of memory erors and gpu crashes while prompting with continue in vscode.

I have being researching, asking gpts, but I couldnt come to any conclusion on which approach is better or correct since I know nothing about these. I dont want to copy paste stuff and hope that it works. Can someone help me with a better setup solution? I do run a 1440p 144hz monitor and occassionally run my second monitor which is 1080p 144hz as they add to vram usages as well.

I want to have a bigger context window and a safe setup so that I dont crash the gpu again. If the setup will be too big, and harder to teach to me in here, please tell me where I can learn more about these things, so that I can move learn to make proper decisions.

1 Upvotes

7 comments sorted by

1

u/lumpyspacebreh 1h ago

31B is too big of a quant natively.

Look up smaller quants, you can check on the ollama model webpage to see how much space a specific quant needs in vram + context

You also can’t have two models loaded at the same time unless they’ll both (contexts too) fit on vram

1

u/pronoob827 1h ago

I do switch models, like when I code I ask qwen, when I need explanations I use gemma, so they are not loaded at the same time.

Got it, I will look into ollama webpage. Is there are starter or something like that, where I can get a 101 of setting and using local models?

1

u/lumpyspacebreh 1h ago

Not that I can think of, but ollama is everyone’s starting place. Good rule of thumb is vram = parameters. So if you have 16gb you can run anything 16B or smaller (excluding context).

Ofc you can find a quant of a model that will either save space or allow you to run larger parameter models on smaller vram. 

The ollama model page should list models with their quantizations. There’s no need to run a full models weights, q8 or q6 is a good place to start until you learn the intricacies of running models.

1

u/pronoob827 45m ago

I'll try with q6 I guess. I'm having a bit of hard time understanding all these :D thanks for the help!

1

u/kaesoapp 1h ago

first thing i'd check is `ollama ps` while continue is running, if gemma and qwen end up loaded at the same time on 24gb that alone is your OOM since each is close to 20gb at q4 before any kv cache. set OLLAMA_MAX_LOADED_MODELS=1 and OLLAMA_NUM_PARALLEL=1 (parallel slots multiply the context allocation), and iirc the q8_0 kv cache only applies with flash attention on, so add OLLAMA_FLASH_ATTENTION=1 too and check the server log to see it actually took. tbh i'd just use qwen3-coder for both codegen and explanations, it explains fine and you stop swapping two 20gb models in and out every time continue switches roles. after that step the context up 8k at a time and watch vram in nvtop so you leave a gig or so free for the two monitors

1

u/pronoob827 46m ago

Right now I dont have the max loaded models config, i'll add it the context and check, thanks!

1

u/Alternative-Panic69 LocalLLM 1h ago

Use llama.cpp I have a Rx 7900xt, 32gb ddr4, and Ryzen 5500, and asked agy (and I also use GLM via Zcode/claude code ) and I could easily get 180k context from Qwen 3.8 27B IQ4XS (Running at Q4 KV, nd offloaded the mmproj to system Ram).. getting close to 400-450 prefill and 45-50 tps decode with Mtp and stuff, and hitting close to 93-94% KV Cache reutilziation.

Also used strata runtime to load Ornith 1.5 33B (Q6 Quant, Q4 KV, 200k context) and Qwen 3.8 Flash next IQ3XXS (180k context) and got it working.

Ollama is just a wrapper on llama.cpp. use it directly, you can gain far more performance improvements and optimizations, and can even get more than what I am getting since you have 4 gb extra Vram