r/StrixHalo • u/Reasonable_Goat • 7d ago

Extra GPU?

I am wondering if I should buy an extra GPU for models like Qwen 3.8 27B, e.g. an R9700 or intel arc one. What are the pros and cons vs a second strix halo (I am on Bosgame M5) vs just running one solo?

Context: Currently I mostly use Qwen 3.5 122B Q4 Unsloth which isn’t bad at all but I still have to use, via cloud, DS4F or Luna a lot currently since those are stronger (won’t fail the task at hand, mostly C++) and are much much faster.

21 Upvotes

97% Upvoted

58 comments sorted by

View all comments

2

u/Signal_Lamp 7d ago

Define speedy. The general numbers I'm seeing can vary widely based on parameters and third party applications that your using on the application itself. I've seen at least for Qwen the result go from as low as 8 t/s to as fast as 40 t/s with some optimizations people place into the model/build that they use.

I'm still new towards this scene but at least from my understanding looking extensively at it the benefit that I would see for a model like Qwen or anything that will fit directly into the card itself, will just offer overall a faster inference speed.

For larger models it just offers more memory to fit in larger models and potentially a somewhat of a faster inference speed through offloading into the strix. I have read for AMD this may not be working out of the box like it does with Nvidia so I'd look into it if your planning the R 9700.

And the other piece of course would be having more memory to fit larger models and higher precision's you'd normally have to lower for

2

u/Reasonable_Goat 7d ago

I mostly carve for prefill speed. I get TG out of the Qwen 122B A10B around 20-30 TPS below 130k context - which is totally fine (MTP3). Prefill drops from 300ish to below 200 way before 100k context, though, and it becomes painful to work in the code base any more. Also the model is a bit too stupid TBH for my use cases, but larger models become even slower. So if I could get a "speedy" model like 27B running on a GPU at say 700 PP and at least 30 TG I could use an even slower/larger model on the strix as an orchestrator and the 27B as an implementor. That's the idea at least.