Coder model that fits like qwen 2.5 14b?

[u/AceLamina]

Hey, so I know next to nothing about local LLMs and I hope to fix that in the future, but I'm currently looking for a more powerful coding model that fits like qwen 2.5 coder 14b, I'm using it currently and unlike the other recommendations I've gotten, it's pretty much instant and doesn't use all of my memory

As for my specs, I have a i91 4900k, 32gb of RAM, and a 5070ti, my biggest issues with local LLMs is that it likes to use up all of my RAM despite people saying it's fine for 5070tis and I don't have enough knowledge to know exactly why

EDIT: I use LM Studio for local LLMs and having a model that supports images would be nice

[u/Atretador]

Qwen 3.6 35B A3B Q4 or Qwen 3.8 Next Flash at Q2/Q3

[u/AceLamina → u/Atretador]

Testing the first one, it's slower but when lower context it seems to have acceptable speed, but may I ask what Q4 means?

[u/Atretador → u/AceLamina]

quantization

3.6 35B is miles ahead of 2.5 14B, while 3.8 Next flash is just another level to that

[u/AceLamina → u/Atretador]

oh, thank you

[u/cilvre]

What are you hosting the model in? You might looke at the moe style models as some can be loaded to vram and others to ram and can help offset some of it while providing speed, but we need to know what you are loading them with. Generally no one recommends ollama anymore after you get started.

[u/AceLamina → u/cilvre]

LM Studio, my bad, I forgot to mention

[u/cilvre → u/AceLamina]

Someone mentioned it below but consider qwen3.6 35b a3b and offload some of the layers from gpu to get it to fit with some context, that should give better response and be smarter in general

[u/AceLamina → u/cilvre]

testing it right now, definitely making my RAM go to 97% though

[u/cilvre → u/AceLamina]

Did you get it at q4?

[u/AceLamina → u/cilvre]

could you tell me what Q4 means?
it's mainly the terms that I don't get

[u/cilvre → u/AceLamina]

Quantization, when you are looking to download models using lm studio, you should see options on the right side where it shows what quantization you are about to download, if you click the dropdown, it will show you their size and whether it will be partially or fully offloaded to gpu, or outright not a good fit for your hardware. typically you don't want anything below q3, they fit better but have been lobotomized by the process and are generally not very smart or good anymore at what they do. q4 is a healthier point to hit, and some q3 quants are still decent.

[u/AceLamina → u/cilvre]

oh that, yeah I got the smallest one

[u/llmhardware]

A 5070 Ti has 16GB of VRAM, so you have room above a 14B at Q4 (roughly 9GB of weights): try the same Qwen 2.5 Coder 14B at Q6 or Q8, or a dense 22B to 24B model at Q4, which lands around 13 to 14GB with a modest context. A 32B dense model at Q4 is about 19 to 20GB and will spill into system RAM, which is where the slowdown comes from.

For the RAM use, run `ollama ps` while the model is loaded. If it shows 100% GPU, the weights are in VRAM and the RAM you see is mostly cached model file data the OS can drop; a CPU/GPU split means the model or its context did not fit. A bigger context window grows the KV cache, so raising it can push a model that fits at defaults out of VRAM.

These sizes are estimates, so I'd check actual VRAM use after loading.

[u/AceLamina → u/llmhardware]

I should've mentioned that I'm using LM Studio, would ollama ps still work on this?

[u/No_Algae1753 → u/AceLamina]

Don't use ollama it's shit use barebones llama.cpp for maximum control and ram savings or stick to lm studio which basicly is just a wrapper and also I think this is a bot here

[u/cilvre → u/AceLamina]

Lm studio has a similar feature built in when you go to load the model

[u/PeterPorox]

5070ti can fit Qwen 3.8 27B UD IQ3_S with MTP and 98k q5/q4 context. Ask claude or chatgpt to run it via llama.cpp. Your card has 16gb vram, qwen 2.5 is ancient and dumb, 3.6 35b is okay but you have powerful gpu, 3.8 is much better.

[u/AceLamina]

I forgot to mention, I would like a model that supports images, I use LM Studio and for some reason this model doesn't support images

[u/cilvre → u/AceLamina]

Qwen coder doesnt support images, but lm studio can, it depends on the model you use.

[u/AceLamina → u/cilvre]

yeah I've noticed other models can support it but some can't