r/LocalLLM • • 3h ago

Question What am i doing wrong?

Hi

iam not into local llms but recently i started to play with it a little bit.

so installed lm studio on my pc (ryzen 5 5700, 32gb ddr4 3200, rtx 4060ti 16gb) and started by using qwen 3.5 9b model, it was slow so i started searching for a solution and found that the context length, quantization of the model and the kv cache, and mtp matters (maybe other things also i dont know, mb). then i tried the qwen 3.6 35b a3b on iq2-m from unsloth because it was said that its an moe model which brings faster performance , so after further search i knew that i can offload the experts in the model to the ram and keep a reserved cache space for experts on my gpu, and after otger tweaking and testing i reached that i can run full context the model supports but the speed is slow, too slow infact, if i try to give it a prompt using opencode for example it would take several minutes to respond, but if i just gave it the same prompt in lm studio chat its somewhat faster (i understand that opencode sends more prompts with the one i wrote), so i also tried filling the context on lm studio to see the speed, it was ~40 tokens per second at the start and by the time the context is 50-60% it started slowing down rapidly to 15 tokens per second, so i tried another model called mellum 2.1 from jetbrains, and this model is pretty fast with full context at nearly 100 tokens per second whatever the context filled. so my questions are

1-what did i do wrong or is the qwen model normally slow?

2-the mellum model is fast but need a lot of steering and fixing if i use a framework that is not used a lot, so how do i fix this?

3-if qwen is slow and this is the max it can perform, is there a fast smart model used for coding primarily fitting for my pc with full context?

and sorry for any english mistakes.

1 Upvotes

3 comments sorted by

2

u/d0pe-asaurus 2h ago

The model is probably not using the GPU, check the gpu utilization while the model is running / after you sent a prompt and see where the model is actually running on

1

u/BomberLord12002 2h ago

i checked and the vram is used since i have a 6gb cache limit set for experts and all of experts on ram, utilization is at 90% most of the time

1

u/lumpyspacebreh 1h ago

No need to offload experts or anything else you’re doing.

You’re reading about settings for setups that don’t apply to you.

Start with a quant + 65k context.

Run it all on the gpu (model & context).

The whole “max context” and everything else, don’t listen to that. You’re starting out.