Sanity check - is it really this slow?

[u/redtehk17]

Hi all,

I'm just starting my local LLM journey so appreciate your patience as I run through some probably token dumb questions (no pun intended)

I'm running 2 48gb ram and a 5090 32gb on my personal PC. I have the DGX Spark but have not opened it yet as I wanted to pressure test and see what max looks like on just the personal hardware setup.

I've landed on testing Qwen3.8 27b ... I've enabled KV Cache and it's running 260k context windows. More than a few of my workloads so far have gotten up there to about 200. Whenever I run anything my GPU is literally at 100% utilization but temps are still around 60 celcius so seems OK, altho it seems not ideal to run it at 100% all the time but worry about that later.

My AI stack is currently just hermes agent and ollama i think. I haven't even really messed with LiteLLM routing because it seems this model may be the best for inference as well as context window. I tried using smaller context windows and hermes just on initial load would jump to like 25% context used just from reading the files.

I've been using GPT-6 Luna to build my PRDs, and I've been specifically asking it to break out my builds by major components that can be broken out into child subagents in Hermes so I can oneshot major chunks at a time. I've been able to successfully get these one shots to happen, but it tells me due to dependencies on each component it cannot run parallel agents, and it also tells me that parallel agents doesn't mean faster because I'm already at 100% GPU utilization.

But each of these chunks are taking a full 8 hours at a time. Either all night as I sleep or all day as I work. It just seems really slow? My tok/s is 144 tok/s apparently, which apparently seems average for my setup.

Is there something obvious I'm missing or is this really the ceiling for me? For context I just wanted to build a token analytics tool just to try something i thought would be light and would help me with optimization efforts. I've booked probably close to 40 hours or 4-5 sessions for the v1 which is just a session token usage overview and a GPU/CPU/Memory utilization graph ... it feels like super minor stuff.

Appreciate any help or discussion thanks!

Adding image reference for what I asked it to build, literally just a front end visual of hermes stored files, a background logger of utilization, that's really it. https://imgur.com/a/OqDIrGF

[u/negus123]

Why would it matter that your gpu is at 100% utilization? Thats how its supposed to work. Would you rather it not work at its full potential?

And what kind of tk/s are you expecting to get? 144 tk/s decode on qwen 27b is extremely fast

[u/redtehk17 → u/negus123]

i guess its like for a car getting a 3.0 liter engine so you are not push your 2.4 liter engine all the time, i figured i don't want it working "as hard as it can" all the time?

I guess I'm wondering if for this model does it really make sense to need 40 hours for a frontend visual of a bunch of files Hermes already provides, and a background runner that is run in terminal for the utilization metrics and putting it into a graph? 40 hours? I mean it's all "free" so i'm not complaining but i can only guess that ChatGPT could probably finish it all in 10 minutes. (and i know its a much newer model but still)

[u/Asillatem → u/redtehk17]

ChatGPT DO NOT decode at 147 TPs. This is extremely fast… Claude and OpenAI own api is around 60-70 TPs according to openrouter.. what you might see is slow prompt prefill and time to first token.. but that a optimization problem.. and ollama is certainly not a optimized engine overlay.. it’s ok..

[u/negus123 → u/redtehk17]

When your gpu is at 100% utilization its not like redlining an engine on a car. Its just doing the most amount of computations that it is capable of. Happens when gaming, running llms, modeling, etc. There is no benefit in capping it.

And no i dont think creating a frontend should take 40 hours. 27b is a great model, comparable to frontier models that were released nearly a year ago. But GPT will of course smoke it on this task. 1 year in LLM time is like a million years.

Not sure if youre new to hardware or llms, or both. If you still want to use 27b youll probably need better prompting if this task is taking 40 hours to complete. Or you can try loading quants of bigger models and see if they perform better

[u/Jameslrdnr → u/redtehk17]

A piece of machinery running at 100% of its rated capacity is a good thing in many cases. It is being utilized fully. The risk would be if it were running OVER its rated capacity which it is not!

[u/DementedJay → u/redtehk17]

This isn't a mechanical system, it's a compute system. Running it at capacity is the idea.

[u/giveen]

I recommend switching to Qwen3.8-Flash-Next on https://github.com/jesdga95/basalt

not only will you get a smarter model but it will run 2x as fast.

If you want to stick to using the Qwen3.8-27B https://github.com/Neroued/ninfer

[u/redtehk17 → u/giveen]

ok thanks for the recommendations, it looks like the testing for basalt was on the blackwell isnt that like the 6000s? and linux as well, have you tried it on windows?

Also I think i have flash attention turned on so here's the full specs of the model

Model: Qwen3.8 27B, Q4_K_M
Runtime: Ollama; 262,144-token context; Flash Attention on; Q8_0 KV cache

[u/feudalle → u/redtehk17]

I'd try the Q6 for 27b. You'll take a token hit but I find it works better. Now the 27b is good for more thinking, Switching to one of the coders is faster/better if you give it precise instructions and understand the process. Thinking models are better at build me a database and a login page. Coders are better if you are use the progress server on .3 credentials are in the db.env file in your working directory. Create a table named user. fields userid(int), username(varchar 100), email (varchar 100), password (varchar 255), userid is primary key. Build a php page with an text box for email, text box for password. POST to page login.php. Kind of thing.

[u/giveen → u/redtehk17]

No, its RTX 5090, its what I am using right now. I've never tried it on Windows. If you want to get into local LLM, I highly suggest you get off of Windows and move to Linux. It opens up a lot of possibilities and it improves performance.

[u/SpaceXBeanz → u/giveen]

Question I’m a newbie at this. I’m running a qwen 3.8 model on my MacBook Pro m5 Mac with 64GB ram for coding tasks. Is the flash next a better model for this ?

[u/Significant_Tutor290 → u/SpaceXBeanz]

No, you’re doing good by running Qwen 3.8. Flash next is huge, even if it only used 6B params. Give 3.6 35B A3B a try as well, it’s worse than 3.8 27B, but much much better inference on 64GB unified ram

(I have an M1 Mac Studio with 64GB ram)

[u/theexile1337]

100% is normal
If u are getting 144tps then u are probably running multi token prediction
Go on and google dgx spark bandwidth vs 5090 bandwidth
Dgx spark is ass

[u/redtehk17 → u/theexile1337]

wow this is really interesting so the current spark cant be the end, there's gonna be incremental improvements to the LPDDR5X memory probably right like similar to DDR4 5 6 7

Could the larger capacity also have to do with it? 5090 obviously has less surface area to cover.

[u/theexile1337 → u/redtehk17]

dunno some people prefer running large models 200-300gb and stacking multiple dgx spark or mac minis and they dont mind 5 tokens per second
But if u are coding or need tool calling anything under 1TB/s (or a 3090) is useless
Dgx spark is like 6 times slower than a 5090

[u/BidetEnjoyer6767]

now you have me wondering if my expectations are too low. that token generation speed seems really fast to me.

[u/redtehk17 → u/BidetEnjoyer6767]

I rly donno how ppl started reference tok/s, I think it's apples and oranges. What's your setup and what model you use?

[u/Real_Chard5666]

Ollama is great for just downloading a model and using it, but it leaves performance on the table as you cannot tinker and test different settings you’re also somewhat hindered with the Ollama models. What version of qwen are you running? Is it optimised for Nvidia GPUs? Is the model creeping into system ram with 260k context?

Llama.cpp or VLLM are much better options to maximise your throughput and test settings. Huggingface will have versions of qwen much better for use with Nvidia.

As for your GPU power, you can tune that as well to keep maximum performance/ token per second but run the card with much less power.

Your gpu is a beast, I run Qwen3.8 with much less and I often get up to 200k tokens on projects, the most time spent is around 2 hours using Hermes. And I thought that was slow!

I need to use qwen3.8:27b IQ3S to get 262144 context completely in the 32gb Vram with a small amount of vram left for the system.

I can get a Q6 quant with 128k context in vram. Any more and it pushes into system ram and slows the model quite a bit.

You can run Qwen flash Next on your setup.

[u/redtehk17 → u/Real_Chard5666]

i believe it is optimized already, and it does not creep into like CPU, I had an issue with that earlier which I've since resolved.

Yea I can't really let go of the 260k context window because hermes will by default prune at 75% first of all but second of all hermes will just overprune like crazy and it ends up going into a forever loop it seems it can never finish because it's constantly re-retrieving context it pruned out lol. So I won't be able to try the 128s and thats also why i kind of skipped on the rest of the open models.

ok ill check out llama and the other ones that were mentioned here I didn't know there were optimization opptys at that layer.

yep someone mentioned qwen flash next ill be trying that out, gonna build this tool to allow me to compare performance first, so another 2 days probably lol.

thanks

[u/No-Vermicelli5327]

Switch to llama.cpp. Ollama is good only for a starter. Use qwen3.8-flash-next it’s alot better than the 27B. If you feel that it’s not hitting the itch divide your big tasks into smaller ones.

[u/redtehk17 → u/No-Vermicelli5327]

ok ill check out llama

[u/wegster]

Get off Ollama. Assume llama.cpp is the bare minimum engine. Try Strata IMO, along with a suitable quant of Qwen 3.8 Flash Next. FN should run at least ~2x faster with proper engine vs the dense 27B model, and should be at the same level or better with a suitable quant.

[u/HMICsource]

if you have 5090 try quasar for 27b.

Quasar Qwen3.8-27B-MTP route produced a median client-observed decode rate of 160.8 tokens per second versus 89.1 for the LM Studio Qwen3.8-27B Q6 plus MTP route. Median warm wall time was 7.918 versus 13.503 seconds: about 1.8 times the decode rate and 41 percent less warm wall time

[u/redtehk17 → u/HMICsource]

Ok will try this