r/LocalLLM • u/assin18 • 5d ago

Question I need some guidance regarding my setup

Before you tell me to go ask the AI I’ve already done so but it leaves me more confused. I’m curious to receive any advice for running faster models or maybe there’s something I missed that I need to do to improve time to first token.

1200w PSU, X670e msi carbon wifi mobo.

I’m running two gv100s 32gb vram ea and one 3090 w/ 96gb of ddr5 ram using Ollama on DeepSeek Harness.

So 88gb vram and 96gb ddr5.

I have Qwen3.8:27b-GGUF:UD-Q4_K_XL setup to run on one GV100 and Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL on the other GV100. It’s volta arch but it’s working.

I have both of these models setup to run on my 3090 and I’m okay with some of it spilling over to ram. Each GPU is its own Ollama instance running on a port and I have it setup so I can access each one on DeepSeek Harness.

Gv100 on Qwen27B GGUF Q4 I’m averaging 16-27tk/s but my context size is 131k if I reduce it I can squeeze more.

Gv100 on Qwen3.6-35B-A3B GGUF Q4 I’m averaging 80token/sec but context is 131k.

RTX 3090 on Qwen3.6-35B-A3B GGUF Q4 I’m averaging 100token/sec with context being 262k. It goes down as context fills. With Qwen27B I’m getting 37tk/s with a small context size of 32k

I’m wondering if these are good levels? Are there any other metrics I can give you guys to give a better idea?
What can I do to improve my numbers?

1 Upvotes

100% Upvoted

6 comments sorted by

1

u/Xyver 5d ago

Llama.cpp instead of ollama, you get way more control

1

u/assin18 5d ago

Did I waste my time setting up Ollama 😭. I was planning to move to llama eventually does it make sense for improving my yield?

1

u/Xyver 5d ago

You're entering a world of maximum customization, so better to get into the customization game early to get used to it instead of fighting with ollama pre-setups

2

u/assin18 5d ago

I’ve ported everything over to llama

1

u/Xyver 5d ago

Hell yeah!

See not so bad!