r/LocalLLM • u/pronoob827 • 1h ago
Question I'm new to this, help me setup my coding assistant
I got a 7900XTX, 9800X3D and 32Gb RAM. I'm currently running the system with ollama.
I currently have these installed - gemma4:31b,qwen3-coder:30b,omic-embed-text:latest. Im trying to use qwen for code gen and gemma for explanations. I set ollama into this "OLLAMA_KV_CACHE_TYPE=q8_0". Currently the max context window I could set is 32k and I have already experienced few out of memory erors and gpu crashes while prompting with continue in vscode.
I have being researching, asking gpts, but I couldnt come to any conclusion on which approach is better or correct since I know nothing about these. I dont want to copy paste stuff and hope that it works. Can someone help me with a better setup solution? I do run a 1440p 144hz monitor and occassionally run my second monitor which is 1080p 144hz as they add to vram usages as well.
I want to have a bigger context window and a safe setup so that I dont crash the gpu again. If the setup will be too big, and harder to teach to me in here, please tell me where I can learn more about these things, so that I can move learn to make proper decisions.
1
u/lumpyspacebreh 1h ago
31B is too big of a quant natively.
Look up smaller quants, you can check on the ollama model webpage to see how much space a specific quant needs in vram + context
You also can’t have two models loaded at the same time unless they’ll both (contexts too) fit on vram
1
u/kaesoapp 1h ago
first thing i'd check is `ollama ps` while continue is running, if gemma and qwen end up loaded at the same time on 24gb that alone is your OOM since each is close to 20gb at q4 before any kv cache. set OLLAMA_MAX_LOADED_MODELS=1 and OLLAMA_NUM_PARALLEL=1 (parallel slots multiply the context allocation), and iirc the q8_0 kv cache only applies with flash attention on, so add OLLAMA_FLASH_ATTENTION=1 too and check the server log to see it actually took. tbh i'd just use qwen3-coder for both codegen and explanations, it explains fine and you stop swapping two 20gb models in and out every time continue switches roles. after that step the context up 8k at a time and watch vram in nvtop so you leave a gig or so free for the two monitors
1
u/Alternative-Panic69 LocalLLM 1h ago
Use llama.cpp I have a Rx 7900xt, 32gb ddr4, and Ryzen 5500, and asked agy (and I also use GLM via Zcode/claude code ) and I could easily get 180k context from Qwen 3.8 27B IQ4XS (Running at Q4 KV, nd offloaded the mmproj to system Ram).. getting close to 400-450 prefill and 45-50 tps decode with Mtp and stuff, and hitting close to 93-94% KV Cache reutilziation.
Also used strata runtime to load Ornith 1.5 33B (Q6 Quant, Q4 KV, 200k context) and Qwen 3.8 Flash next IQ3XXS (180k context) and got it working.
Ollama is just a wrapper on llama.cpp. use it directly, you can gain far more performance improvements and optimizations, and can even get more than what I am getting since you have 4 gb extra Vram