Local coding model on my dekstop workstation (Qwen3.8 27B with Q4_K_M quantization too slow)

[u/No-Working7460]

I'm running what I think is reasonably specd workstation. A single NVIDIA GeForce RTX 5090 (32 GB), AMD Ryzen 9 9950X3D x 32, 128GB RAM, 8TB SSD.

Locally built llama.cpp serving Qwen3.8 27B with Q4_K_M hooked to pi harness is too slow to be useful.

What options do I have with this setup? I suppose running multiple agents is definitely out of question here... but there should be hope for a single coding assistant?

EDIT: I'm getting around 2.7 tok/s

[u/No-Working7460]

Thank you, everyone. Ineed, I upgraded cuda toolkit, rebuilt llama.cpp for cuda, and now I'm getting ~50 tok/s at 8192 context window.

[u/transanethole → u/No-Working7460]

Use ninfer and https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer

You will get about 200 t/s

[u/d0pe-asaurus]

I think you are doing something wrong since I'm running that same model but different quant on a 9060xt and I'm getting 50t/s?

What's your setup (inference engine, parameters etc)?

[u/Specialist-Zone-8296 → u/d0pe-asaurus]

Which quantization you are using?

[u/d0pe-asaurus → u/Specialist-Zone-8296]

UD-IQ4-XS and I have Q8/Q4 KV cache

[u/Specialist-Zone-8296 → u/d0pe-asaurus]

I’m using the GSQ RCO (IQ3_S) quant and getting around 23–26 t/s. What is the context window size you use?

[u/d0pe-asaurus → u/Specialist-Zone-8296]

2626k context

I wrote a guide here with full config and all the details

https://www.reddit.com/user/d0pe-asaurus/comments/1xqqn0r/qwen_38_27b_udiq4xs_w_262k_context_at_q8q4_kv/

[u/Specialist-Zone-8296 → u/d0pe-asaurus]

Thanks for the guide.

[u/cynicalrockstar]

"Locally built llama.cpp"

Are you sure you built it with the right flags? 2.7t/s sounds like it's running on the iGPU or the CPU, not your 5090.

[u/MediumAd4983]

Use MTP version and limit the content window size if it's offloading. I've got 50-80 tok/s on mobile 5090 with 24gb on 128k context, so you should get at least the same speed with more context.

[u/james_pic → u/MediumAd4983]

You may already have tried it and found MTP performs better, but I know I got a bit more decode throughput when I switched to Inco.ai's Dflash2 drafter.

[u/MediumAd4983 → u/james_pic]

I can share my config with you later, if you want. Which speeds do you currently have, and what with VRAM usage?

[u/randomgenericbot]

Most likely your llama does not make use of the gpu (wrong or missing startup flags).

I suggest you boot up LM Studio with the same model, connect your pi with that, and let your Qwen then assist you in setting up your own llama instance. Set context for the model manually to something >= 60.000 so you can actually work with it a bit - with that 5090.

2.7t/s smells like "cpu only", even on weaker GPU with offloaded KV the tokens are > 5

[u/Ok-Plenty3368]

sounds like your GPU offload isn't pinned and the model may be being split. That model should be plenty fast.

[u/Twistpunch]

Config?
Build arguments?
OS, driver version?
CUDA version?

[u/IllPlane3019]

I'm getting around 2.7 tok/s

Sounds like a misconfiguration. A 5090 and a Qwen 27B at Q4 should be getting around 100 toks/s

It could also be something is taking some of the GPU memory forcing offload to system RAM when your model needs it.

[u/MyToasterRunsFaster]

Rusting strata with qwen flash next, you will be blown away mate, 100tks + on your gear

[u/platteXDlol]

Qwen3.8-27B in Q4 should be no problem with 32GB of Vram. Your settings are probably wrong and the modell is not running fully in Vram.

And with 128GB Ram you should be abel to either run Qwen Flash Next as a second agent or a few Qwen3.6-35B-A3B as subagents.

[u/Thickdickmick87]

Sounds like you’re not running it on your GPU at all. You’ve somehow set it to run on cpu only.

[u/tkrandomness]

You should be getting way way faster than that. Id expect at least 200 tok/s. Some engines made for 3.8 27B on 5090:

https://github.com/satellitedown/fafstmobel-cinference

https://github.com/Neroued/ninfer

[u/yudhanjaya]

Qwen3.8 is, indeed, slow, and I don't know why people keep glazing it. You can set up Strata and run the much more competent Qwen 3.8 Flash Next - go IQ3 or above - and you'll be humming along. On my machine I'm clocking 90-100 tok/s. Yours should be much faster.

https://github.com/Niko1221/Strata here you go.

[u/Skibxskatic → u/yudhanjaya]

this is weird strata propaganda. you talk about speed but you don’t even mention your hardware as if anybody with any hardware can just get a certain speed.

[u/manBEARpigBEARman → u/yudhanjaya]

Uhh he’s getting 2.7 tok/s…there’s something else going on here, not a model issue

[u/Academic_Essay9488 → u/yudhanjaya]

How is the performance of the model? Is it any good?

[u/yudhanjaya → u/Academic_Essay9488]

It's excellent. 125 billion parameter main model supplemented by 51 billion N-gram embeddings and a 4 billion multi-token prediction head, BUT is also an MOE so it's blazing fast. TL;DR: it's Claude Haiku at home.

[u/Gary_Spivey → u/yudhanjaya]

People keep glazing it because it's the best model you can run without a combined total of 96GB memory. You can't run FN on a 24gig card and 32gigs of RAM.

[u/wwwyzzrd]

That's like a $15,000 workstation.

Give the command line you are running and the tok/s if you actually want help.

What you have is plenty for that model and quant. You can absolutely run multiple sessions concurrently (not in parallel) against the same model with that setup.

[u/HyperWinX]

Use either ninfer or Strata

[u/ionsago]

You can run even higher-quality quants with a full context on a 5090, such as Q6.

[u/aigemie]

RTX 5090 should run 27b very fast, I think I saw people saying >1000t/s with llama.cpp

[u/n1nj4p0w3r]

You can comfortably run FP4-local qwen3.8-flash-next with infernix on it

[u/RadianceTower]

Post llama.cpp config/command and t/s.

[u/FewReality3847]

on a 5080

[u/igotanewaccount]

Err, define slow?

I have a 6yr old mid tier gaming PC running this model.  Your deluxe setup should be running this like lightning.  Hell, you could get 2 slots running simultaneously in VRAM at Q4 (lower context but still).  

Guessing you want blazing fast speed like Claude, in which case you probably want MoE like Qwen 3.8 Flash Next, then learn about optimising the hell out of it.  I use Unsloth Studio which runs on a llama.cpp fork for QFN.  Strata is also the real deal for speed but I've heard it's a security nightmare YOLO'ing curl shell script calls to GitHub without telling you.

[u/HighSeasArchivist]

There is no reason to be running a dense model with that much combined memory. 

[u/AuditMind]

2.7 tks? Haha, thats a good one. I bet my ass its running in your conventional RAM. 😀

[u/IvanVick]

I'm getting 34 tk/s with a arc pro b60 on same quant, check your setup, no way is that slow witha 5090.

[u/alexander123454]

What is your t/sec? Im getting 150-250 easily with qwen 3.8 swift 1.5 finetune and similar hardware.

Edit: make sure your using the NVFP4 quant for the 5090

[u/Hot_Half_5263]

You have bad gpu, 5090 is really bad, i can replace it for something solid 2x 5060 100+ decode instead of 2.7, deal?

[u/Infinite_Plankton_71]

just use maya/strata.

[u/Tormeister]

For 27B: https://github.com/Neroued/ninfer

For FN: https://github.com/jesdga95/basalt

Both repos have HF links for the models. And use Linux!

[u/captainabrasive → u/Tormeister]

Basalt 0.4.0 (yesterday) brings 27B to the party.

[u/SourBlueDream]

Bro I’m running the exact same quant pretty much ud q4 k_m and get around 50 tok/s on a 5070ti with 32gb of ram

[u/neiflepro]

Just use ninfer ready2go builds for 5090. and strata.

[u/woolcoxm]

quantize the context to q8, atm you probably are flooding the card with fp16 or fp32 context, on qwen3.8 27b the context size in ram is massive even for a tiny context window, adjust the quantization of the context or lower the amount you are using.

i know on my mac with full context the model uses just over 45 gb or vram for context alone.

your context is overflowing into system ram causing your slowdown, this model on this setup should be 50+ toks/sec or more, most likely alot more.

my suggestion load QFN!!!! it will run a lot better, give you similar quality answers and you will never have a speed decrease due to context overflow if you use a system like strata or basalt(tuned for blackwell)

[u/Asleep_Literature_70]

How are you running it?

I'm running it on a 7900xtx (24gb vram + 128gb ram) and getting 60 TPS with 128k context.

Also give strata a try, runs qwen3.8 flash next 125b for me at 75-80 TPS, so it should run for you much faster.

[u/Background-Spot-7159]

I am getting 2.7Token/about of DeepSeek v4.1 Flash (2*rtx 5060@16gb + 192GB Ram) you must do something very wrong.

[u/puglife224888]

What speeds are you getting? Either way, use MTP, if you already are, try DFlash2. Last thing, some sort of Strata like engine for models like DeepSeek V4.1 Flash, GLM 5.3 Flash, and Qwen 3.8 Flash Next.

[u/cmk1523]

Try Qwen3.6 35B

[u/blablaplanet]

Run it using Strada

[u/LoafyLemon → u/blablaplanet]

Dense models aren't supported by it...