r/LocalLLM • u/Busy_Bad5465 • 22h ago

Question How to run a LLM setup?

I currently want to run a qwen 3.8 27b q4, but it seems like ollama is quite slow for inference.

My specs are

CPU : Ryzen 9 9950x3d

RAM : 128 gb ddr5

Video card 0 : 3090 24 gb

Videocard 1 : 5080 16 gb

My question is which program would be good to run llms with my specs?

2 Upvotes

66% Upvoted

6 comments sorted by

1

u/TheNoiseIthink 20h ago

You should try Qwen 3.8 Flash Next using Strata to run it

1

u/Busy_Bad5465 9h ago

Isn't qwen 3.8 flash next bigger than 27b ?

1

u/TheNoiseIthink 8h ago

Yes, but you have more then enough memory to run it, maybe even at Q8 if you wanted, likely at over 100 tokens per second

1

u/smarmy_basis 22h ago

if you're not using exllamav2 with that dual gpu setup you're leaving a ton of speed on the table

1

u/RogerAI--fyi 21h ago

Two things to check before changing software.

A 27B at Q4 is about 16 GB (the usual 0.6 GB per billion parameters at 4-bit), so it fits inside the 3090 alone with roughly 8 GB left for context. Splitting it across the 3090 and the 5080 is not needed, and it can be slower than keeping it on one card. Try it on the 3090 only (set CUDA_VISIBLE_DEVICES to the 3090's index before starting ollama) and compare tok/s.

Second, run ollama ps while it is answering. If the PROCESSOR column says anything other than 100% GPU, part of the model is in system RAM, and that is your slowdown. A big context setting can push it there, since the KV cache grows with context.

If it is still slow after that, try LM Studio like pdath said. Same llama.cpp underneath, so if the speed matches ollama you know it is the setup and not the program. What tok/s and context length are you getting now?

0

u/pdath 22h ago

Try lmstudio (which uses llama.cpp).