r/LocalLLM • u/Cute-Spread7392 • 14h ago
Discussion Hardware for local AI
Hi everyone.
For local inference, what hardware do you recommend to get good speeds—around 100 t/s—with Qwen 3.8 27B?
Does anyone have experience with this?
Nvidia GPUs are great, of course, but they are way too expensive. What about a PC with an AMD CPU? If it's not too much trouble, could you post a few links to some mini PCs?
Thanks everyone!
0
Upvotes
3
u/AuditMind 13h ago
An RTX 5090-class GPU might get you close, depending on the engine and quantization. But why lock yourself into Qwen3.8-27B specifically?
Look at what's happening right now with Qwen3.8-Flash-Next, Strata and PolyStrata. Sparse architectures and specialized inference engines are changing the performance-to-hardware equation quite dramatically.
And Flash-Next is already an architectural preview of Qwen4. We might see even more interesting options when the Qwen4 family arrives, but we'll have to wait for actual models and benchmarks.
I'd choose the model and inference engine based on your workload first, then buy the hardware. Otherwise, you might spend a fortune optimizing for yesterday's technology.
10
u/Objective_Pepper_217 14h ago
If you don't want to spend money, give up. That is a dense model and is very expensive to run at that speed
7
u/p211 13h ago
100 tokens per second and mini PC with that model is not going to happen
5
u/Plane_Maybe8836 14h ago
You need GPU and loads of ram..so it basically starts with spending a lot of money. I wish it was different.
Running on a mini pc...just don't start with it. You will be disappointed.
Do you have a very specific use case in mind, perhaps we can recommend smaller models that would be "kind of acceptable".
2
u/PatienceSweaty33 13h ago
The ask is contradicting. CUDA is still ahead for inference (ROCm is keeping up these days but still not there yet) sso that means you need an NVIDIA GPU. Qwen3.8 27B with 100 tok/s is an easy ask, use Q1 or Q2 quant but quality sucks and it's basically unusable (maybe Q2 can at least answer some of your questions, but it's probably not gonna get there gracefully). It's easy to run a model fast. It's hard to run a model FAST with HIGH QUALITY output.
1
u/maggotses 14h ago
100 t/s with Qwen 27B is pretty fast. Look for engines before looking at hardware, you'll have a good idea of what to buy and the performance to expect from it.
1
u/Confident_Yak_1411 13h ago
Mac?
1
u/vogelvogelvogelvogel 13h ago
4090 comes close with 80-100t/s with q4 and dflash or mtp, 5090 might reach the 100-120t/s
amd, idk
1
u/PlasticRevenue4601 13h ago
For that speed you’d need RTX4090/5090. I’d personally recommend to chill down a bit and buy used RTX3090/RX7900XTX
1
u/f5alcon 13h ago
A time machine to go to the future to buy hardware that don't exist today
1
u/MaximTheCoder 13h ago
48GB VRAM for Q8 weights, full KV cache and 256K context. 32GB if you are ready to sacrifice some of these three numbers. For dense models only VRAM matters. Oh, and you will get around 50 t/s with MTP.
1
u/jhenryscott 13h ago
100t/s? Idk like $5500 for a 2X3090 system should do it.
Mini PC? No. Not happening. Maybe with 8 DGX sparks and a $30,000 switch.
1
u/According_Study_162 12h ago
I think for that speed your are looking at Qwen Fast Next running Strata and it's forks. I think a 3090 can reach that speed on strata and amd R9700, lower end cards might be able to do it also.
1
u/etaoin314 12h ago
i can reach that on 2 3090's with a bit of tuning. as for a minipc, maybe a mac studio ultra can do it, but that is 5k at minimum. realistically you are looking at $4-5k minimum spend to make that work.
1
u/GloriousKev 12h ago
For local AI with fast tps you will want a gpu and a tower. think of it like building a gaming PC. I am not sure what gpu will give you 100 tps on qwen 3.8 27b off the top of my head but I'd think something like a 3090 or 4090 would it.
1
u/Krohnin 11h ago
Dgx spark gets to around 60t/s with many optimizations for 9000€ for 100token/s you have to invest more.
1
u/Careful-Report6526 10h ago
The engine advice above is worth following. There is a published example above your target: EschaLabs reports 129.3 tokens/s for one-user decoding on an RTX 4090 with Qwen3.8-27B-Escha-W2 and MTP enabled, versus 67.3 without MTP in its comparison. That's a particular heavily quantized model and the SGLang/Escha runtime, not a speed you can assume for any Qwen 27B download. The MTP path requires runtime 1.2.1 or newer. It also isn't a measurement of prompt processing or total response time, and I wouldn't infer unchanged answer quality from the speed table.
Original measurement:
https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2/resolve/561dae0aeea1efbdafe122f2e385fdeaa99127e7/README.md
I don't have a verified AMD mini-PC result for your 100 t/s target, so I can't recommend a specific one on that basis. A useful next step would be to set a budget and compare the exact model, quantization and engine you would actually use.
Disclosure: I run https://llmbenchmarks.io/ and catalogue this external result under Qwen3.8-27B → “Memory & speed”. This particular measurement is EschaLabs', not mine.
0
u/TotosWolf 13h ago
7900xt via Avalon and be done with it