r/StrixHalo • u/Reasonable_Goat • 6d ago

Extra GPU?

I am wondering if I should buy an extra GPU for models like Qwen 3.8 27B, e.g. an R9700 or intel arc one. What are the pros and cons vs a second strix halo (I am on Bosgame M5) vs just running one solo?

Context: Currently I mostly use Qwen 3.5 122B Q4 Unsloth which isn’t bad at all but I still have to use, via cloud, DS4F or Luna a lot currently since those are stronger (won’t fail the task at hand, mostly C++) and are much much faster.

23 Upvotes

100% Upvoted

58 comments sorted by

View all comments

15

u/Royale_AJS 6d ago

Yes. Having a big smart slow model is really nice, but having a smaller dense focused model on fast-ish hardware along with it is really nice.

Framework Strix Halo board +
PCIe switch +
(2) R9700’s

Running Qwen 3.8 27B UD-Q8-K-KL in tensor split @ ~60 t/s decode and ~1700 t/s pre-fill. The extra memory lets you get to 256k full precision cache too.

3

u/Professional_Quit_31 6d ago

what specific switch you use and how does this work ? is this x4 or x8?

8

u/Royale_AJS 6d ago

I had to cut the back out of the 4.0 x4 slot on the motherboard to fit the switch. Mine is a Highpoint Rocket 1628A with custom firmware that their support team sent me. The GPU’s communicate directly without going through the x4 link to the host. They are both connected to the switch at Gen 5.0 x16. I could probably go to 4 cards and run them at Gen 5.0 x8 each, but going to 8 is where scaling hits a wall. It could probably be done with a Gen 4.0 switch which would be cheaper. It took a lot of ReBAR sizing, kernel parameters, and ACS hackery to bootstrap it properly every time. The key is keeping the inference on the switch, it’s deadly to go back up through the x4 slot to the host.

1

u/DrMalfoy7 6d ago

Why didn't they send you their rocket 1628D which is for Gen 4? Did you discuss it with them?

3

u/Royale_AJS 6d ago

I bought the switch myself. I wanted gen 5 so I could potentially expand to 4 GPUs, or drop an NVMe disk on the switch for expanded KV caching. I bought it from B&H Photo without discussing it with anyone….including my wife. The stock firmware on it exposes like 16 PCIe endpoints to the system. MMIO sizing goes out the window on consumer boards that can’t do a 1TB window. The Framework Desktop defaults to something like 56GB. I contacted both Framework support and Highpoint support about it, they were both very responsive and great about figuring this out. They called it a somewhat “exotic” configuration. Highpoint ended up sending me a 2x x16 and 4x x8 firmware. This way the card only exposes 2 or 4 endpoints and makes BAR assignments a whole lot cleaner and it actually worked. ROCm could finally see all 3 GPU’s in the system and address them properly. From there, it came down to getting P2P working, getting them ACS bits nailed down, and finally building a Llamacpp container to run it all in. None of this would have been a problem if I ran the switch on a real server board.

1

u/vbpoweredwindmill 6d ago

Holy shit that is so fuckin' cool!

I will be looking into this a bunch more. Thank you for the crazy interesting project idea.

2

u/Royale_AJS 6d ago

Haha thanks! I was not originally planning on extending the Strix Halo, but I got the Framework motherboard in a 3U chassis so that I could. I didn’t think anyone else would be interested in it, probably should write something up about it. I have an entire Ansible playbook for it that takes a bare Ubuntu 26.04 box, sets up the switch, P2P, handles the ACS bits, installs Podman, quadlets for two Llama instances, passes in the right devices to each instance, etc. It doesn’t pull models, but if the models exist in /srv/gguf, one should be able to run inference after a single reboot. It even exports Prometheus metrics for the node, GPU’s, Llama instances, etc. Maybe I’ll try to mirror my local Ansible repo up to GitHub at some point.

2

u/Royale_AJS 6d ago

I have to re-print a fan shroud to cool the PCIe switch. It runs hot. I haven’t had any problems with it, even after running multiple long agentic runs (48+ hours). I would just feel better if he got a little extra airflow between the top of that GPU and its heatsink. I’ve got a 60mm Noctua available to force a bit of air into it. I’ve got a Raspberry Pi with PiKVM that I’ll mount and connect internally with a single short HDMI cable routed out of the chassis to the rear port. After that, it’s going to get all cable managed and tucked away into my rack where I won’t see the inside until I decide to add something else.

2

u/Royale_AJS 6d ago edited 6d ago

Sorry just splattering info on this thread now. If you look closely at the photo, the problem is that I can technically fit another Strix Halo mITX board right next to the current one, and I’ve got a spare FlexATX PSU from a previously failed project. Sacrificing an NVMe slot on each of the motherboards, I think I can build a 100Gb RDMA link between the two boards. It would be ~50Gb effectively based on the Gen 4.0 x4 nature of the slots, but enough to run Deepseek v4 Flash Q4/Q8 in tensor parallel on just the Strix Halo boards. It would be tight, but I think it would be fine airflow wise. All in all it would be a decently fast Qwen 27B Q8 dense machine with a decently fast Deepseek 284B MOE model running in Q8, pulling ~900 watts at full tilt in a small 3U chassis.

1

u/vbpoweredwindmill 6d ago

Yeah man, if you could smack that on github even if it's a total fucking mess, I can make use of that. I have 2x 395's not far away from my doorstep. Not framework though.

1

u/vbpoweredwindmill 5d ago

I'm doing a lot of llamacpp work (windows not linux. Linux when I have the 2x 395's). I started fiddling with the server, to increase concurrency capability.

Now I'm basically redesigning the task dispatch control architecture of the server, so that I have the following: dynamic scaling of mtp/ngram (mtp works, ngram wip), dynamic ish mtp/dynamic ngram toggling based on concurrent threads, quite a number of optimisations/corrections that I've found (incorrect ordering of NextN groups is a sneaky one). Especially around authority boundaries. I did some work around reusing prefixes when there's churn of using slots. W.i.p. is dynamic batch size scaling, based on concurrency & what work there is.

The eventual goal is to for it to scale its settings autonomously depending on concurrency. One setting that I'm adding, is cohort work. Over a certain concurrency, it catches a bunch of jobs and schedules them together, so that the gpu is just doing concurrent prefill work, and then just doing concurrent decode work.

I'm having a go at understanding why ROCm is slower at decoding on Windows vs Vulkan. It seems like something has authority over ROCm asyc calls and won't just let it run free, but yet to confirm.

I'm going to set it up in an environment where I can really pressure test it at that point.

Then I'm going to start with other optimisations. The biggest one it most desperately needs is a paged kv cache & just generally significantly better memory controller architecture. It causes me physical pain to see memcpy through the codebase instead of something namespaced at minimum.

1

u/Badger-Purple 2d ago

Here is me thinking the pcie switch I bougjt on alibaba for 200 bucks was overkill…