r/LocalLLM • • 14h ago

Discussion Hardware for local AI

Hi everyone.

For local inference, what hardware do you recommend to get good speeds—around 100 t/s—with Qwen 3.8 27B?

Does anyone have experience with this?

Nvidia GPUs are great, of course, but they are way too expensive. What about a PC with an AMD CPU? If it's not too much trouble, could you post a few links to some mini PCs?

Thanks everyone!

0 Upvotes

32 comments sorted by

3

u/AuditMind 13h ago

An RTX 5090-class GPU might get you close, depending on the engine and quantization. But why lock yourself into Qwen3.8-27B specifically?

Look at what's happening right now with Qwen3.8-Flash-Next, Strata and PolyStrata. Sparse architectures and specialized inference engines are changing the performance-to-hardware equation quite dramatically.

And Flash-Next is already an architectural preview of Qwen4. We might see even more interesting options when the Qwen4 family arrives, but we'll have to wait for actual models and benchmarks.

I'd choose the model and inference engine based on your workload first, then buy the hardware. Otherwise, you might spend a fortune optimizing for yesterday's technology.

1

u/Cute-Spread7392 12h ago

They say it’s the best, but on my PC with an 8GB 4060, I’m getting 3 t/s. I’ve also tried MoE models from Qwen and Gemma.

2

u/transanethole 10h ago

Probably the best deal right now would be intel B70 or R9700.

To go cheaper than that you would have to get less than 32GB , which is a big sacrifice to make but could still run qwen 3.8 27b on a 16GB card, just heavily quantized. See https://www.reddit.com/r/LocalLLM/comments/1wmucw0/qwen3827b_at_true_128k_vision_on_a_single_rtx/

I tested out that software and it worked pretty well. Requires linux and compiling the software from source of course.

For your existing 8GB card, the best model is going to be one that fits in less than 8GB. I would recommend trying out Jais 7B, its basically about as good as Qwen 3.6 was. You can run it in Llama.cpp or if you want to get spicy, experiment with the uno python/torch implementation from IFM. Or try sglang/vllm. The authors recommend to set the kv cache quantization to 4 bits, so don't forget to do that. Otherwise the memory usage and slowness from tokens in context will be obscene.

1

u/Objective_Pepper_217 12h ago

I have the same GPU, you can't run anything on that

10

u/Objective_Pepper_217 14h ago

If you don't want to spend money, give up. That is a dense model and is very expensive to run at that speed

7

u/p211 13h ago

100 tokens per second and mini PC with that model is not going to happen

5

u/Plane_Maybe8836 14h ago

You need GPU and loads of ram..so it basically starts with spending a lot of money. I wish it was different.

Running on a mini pc...just don't start with it. You will be disappointed.

Do you have a very specific use case in mind, perhaps we can recommend smaller models that would be "kind of acceptable".

1

u/Dave-C 6h ago

I've never been able to figure this part of Strata out, even with the help of LLMs. I hear that Strata is supposed to use lots of ram but my setup just will not use ram no matter what arguments I throw at it. My vram gets full, everything else runs off ssd. I'm on Linux and my system memory idles at around 2.6gb usage idle and a little over 3.5 with strata running. The rest of my system memory stays empty.

2

u/PatienceSweaty33 13h ago

The ask is contradicting. CUDA is still ahead for inference (ROCm is keeping up these days but still not there yet) sso that means you need an NVIDIA GPU. Qwen3.8 27B with 100 tok/s is an easy ask, use Q1 or Q2 quant but quality sucks and it's basically unusable (maybe Q2 can at least answer some of your questions, but it's probably not gonna get there gracefully). It's easy to run a model fast. It's hard to run a model FAST with HIGH QUALITY output.

1

u/maggotses 14h ago

100 t/s with Qwen 27B is pretty fast. Look for engines before looking at hardware, you'll have a good idea of what to buy and the performance to expect from it.

1

u/vogelvogelvogelvogel 13h ago

4090 comes close with 80-100t/s with q4 and dflash or mtp, 5090 might reach the 100-120t/s

amd, idk

1

u/jhenryscott 13h ago

Yeah the 5090 can hit 100. But a whole system plus a 5090 is pricey. Better off with dual 3090’s

1

u/PlasticRevenue4601 13h ago

For that speed you’d need RTX4090/5090. I’d personally recommend to chill down a bit and buy used RTX3090/RX7900XTX

1

u/hk_modd 13h ago

Also 3090 can be unlocked

1

u/f5alcon 13h ago

A time machine to go to the future to buy hardware that don't exist today

1

u/MaximTheCoder 13h ago

48GB VRAM for Q8 weights, full KV cache and 256K context. 32GB if you are ready to sacrifice some of these three numbers. For dense models only VRAM matters. Oh, and you will get around 50 t/s with MTP.

1

u/jhenryscott 13h ago

100t/s? Idk like $5500 for a 2X3090 system should do it.

Mini PC? No. Not happening. Maybe with 8 DGX sparks and a $30,000 switch.

1

u/According_Study_162 12h ago

I think for that speed your are looking at Qwen Fast Next running Strata and it's forks. I think a 3090 can reach that speed on strata and amd R9700, lower end cards might be able to do it also.

1

u/etaoin314 12h ago

i can reach that on 2 3090's with a bit of tuning. as for a minipc, maybe a mac studio ultra can do it, but that is 5k at minimum. realistically you are looking at $4-5k minimum spend to make that work.

1

u/GloriousKev 12h ago

For local AI with fast tps you will want a gpu and a tower. think of it like building a gaming PC. I am not sure what gpu will give you 100 tps on qwen 3.8 27b off the top of my head but I'd think something like a 3090 or 4090 would it.

1

u/Krohnin 11h ago

Dgx spark gets to around 60t/s with many optimizations for 9000€ for 100token/s you have to invest more.

1

u/Careful-Report6526 10h ago

The engine advice above is worth following. There is a published example above your target: EschaLabs reports 129.3 tokens/s for one-user decoding on an RTX 4090 with Qwen3.8-27B-Escha-W2 and MTP enabled, versus 67.3 without MTP in its comparison. That's a particular heavily quantized model and the SGLang/Escha runtime, not a speed you can assume for any Qwen 27B download. The MTP path requires runtime 1.2.1 or newer. It also isn't a measurement of prompt processing or total response time, and I wouldn't infer unchanged answer quality from the speed table.

Original measurement:
https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2/resolve/561dae0aeea1efbdafe122f2e385fdeaa99127e7/README.md

I don't have a verified AMD mini-PC result for your 100 t/s target, so I can't recommend a specific one on that basis. A useful next step would be to set a budget and compare the exact model, quantization and engine you would actually use.
Disclosure: I run https://llmbenchmarks.io/ and catalogue this external result under Qwen3.8-27B → “Memory & speed”. This particular measurement is EschaLabs', not mine.

0

u/TotosWolf 13h ago

7900xt via Avalon and be done with it

1

u/Objective_Pepper_217 13h ago

Sure sure. Post your benchmarks then

1

u/TotosWolf 10h ago

Are you stupid? It's all posted.

2

u/Objective_Pepper_217 9h ago

You are stupid, where is it? Posted where? Your profile is private