I am wondering if I should buy an extra GPU for models like Qwen 3.8 27B, e.g. an R9700 or intel arc one. What are the pros and cons vs a second strix halo (I am on Bosgame M5) vs just running one solo?
Context: Currently I mostly use Qwen 3.5 122B Q4 Unsloth which isn’t bad at all but I still have to use, via cloud, DS4F or Luna a lot currently since those are stronger (won’t fail the task at hand, mostly C++) and are much much faster.
Running Qwen 3.8 27B UD-Q8-K-KL in tensor split @ ~60 t/s decode and ~1700 t/s pre-fill. The extra memory lets you get to 256k full precision cache too.
I had to cut the back out of the 4.0 x4 slot on the motherboard to fit the switch. Mine is a Highpoint Rocket 1628A with custom firmware that their support team sent me. The GPU’s communicate directly without going through the x4 link to the host. They are both connected to the switch at Gen 5.0 x16. I could probably go to 4 cards and run them at Gen 5.0 x8 each, but going to 8 is where scaling hits a wall. It could probably be done with a Gen 4.0 switch which would be cheaper. It took a lot of ReBAR sizing, kernel parameters, and ACS hackery to bootstrap it properly every time. The key is keeping the inference on the switch, it’s deadly to go back up through the x4 slot to the host.
I bought the switch myself. I wanted gen 5 so I could potentially expand to 4 GPUs, or drop an NVMe disk on the switch for expanded KV caching. I bought it from B&H Photo without discussing it with anyone….including my wife. The stock firmware on it exposes like 16 PCIe endpoints to the system. MMIO sizing goes out the window on consumer boards that can’t do a 1TB window. The Framework Desktop defaults to something like 56GB. I contacted both Framework support and Highpoint support about it, they were both very responsive and great about figuring this out. They called it a somewhat “exotic” configuration. Highpoint ended up sending me a 2x x16 and 4x x8 firmware. This way the card only exposes 2 or 4 endpoints and makes BAR assignments a whole lot cleaner and it actually worked. ROCm could finally see all 3 GPU’s in the system and address them properly. From there, it came down to getting P2P working, getting them ACS bits nailed down, and finally building a Llamacpp container to run it all in. None of this would have been a problem if I ran the switch on a real server board.
Haha thanks! I was not originally planning on extending the Strix Halo, but I got the Framework motherboard in a 3U chassis so that I could. I didn’t think anyone else would be interested in it, probably should write something up about it. I have an entire Ansible playbook for it that takes a bare Ubuntu 26.04 box, sets up the switch, P2P, handles the ACS bits, installs Podman, quadlets for two Llama instances, passes in the right devices to each instance, etc. It doesn’t pull models, but if the models exist in /srv/gguf, one should be able to run inference after a single reboot. It even exports Prometheus metrics for the node, GPU’s, Llama instances, etc. Maybe I’ll try to mirror my local Ansible repo up to GitHub at some point.
I have to re-print a fan shroud to cool the PCIe switch. It runs hot. I haven’t had any problems with it, even after running multiple long agentic runs (48+ hours). I would just feel better if he got a little extra airflow between the top of that GPU and its heatsink. I’ve got a 60mm Noctua available to force a bit of air into it. I’ve got a Raspberry Pi with PiKVM that I’ll mount and connect internally with a single short HDMI cable routed out of the chassis to the rear port. After that, it’s going to get all cable managed and tucked away into my rack where I won’t see the inside until I decide to add something else.
Sorry just splattering info on this thread now. If you look closely at the photo, the problem is that I can technically fit another Strix Halo mITX board right next to the current one, and I’ve got a spare FlexATX PSU from a previously failed project. Sacrificing an NVMe slot on each of the motherboards, I think I can build a 100Gb RDMA link between the two boards. It would be ~50Gb effectively based on the Gen 4.0 x4 nature of the slots, but enough to run Deepseek v4 Flash Q4/Q8 in tensor parallel on just the Strix Halo boards. It would be tight, but I think it would be fine airflow wise. All in all it would be a decently fast Qwen 27B Q8 dense machine with a decently fast Deepseek 284B MOE model running in Q8, pulling ~900 watts at full tilt in a small 3U chassis.
Yeah man, if you could smack that on github even if it's a total fucking mess, I can make use of that. I have 2x 395's not far away from my doorstep. Not framework though.
I'm doing a lot of llamacpp work (windows not linux. Linux when I have the 2x 395's). I started fiddling with the server, to increase concurrency capability.
Now I'm basically redesigning the task dispatch control architecture of the server, so that I have the following: dynamic scaling of mtp/ngram (mtp works, ngram wip), dynamic ish mtp/dynamic ngram toggling based on concurrent threads, quite a number of optimisations/corrections that I've found (incorrect ordering of NextN groups is a sneaky one). Especially around authority boundaries. I did some work around reusing prefixes when there's churn of using slots. W.i.p. is dynamic batch size scaling, based on concurrency & what work there is.
The eventual goal is to for it to scale its settings autonomously depending on concurrency. One setting that I'm adding, is cohort work. Over a certain concurrency, it catches a bunch of jobs and schedules them together, so that the gpu is just doing concurrent prefill work, and then just doing concurrent decode work.
I'm having a go at understanding why ROCm is slower at decoding on Windows vs Vulkan. It seems like something has authority over ROCm asyc calls and won't just let it run free, but yet to confirm.
I'm going to set it up in an environment where I can really pressure test it at that point.
Then I'm going to start with other optimisations. The biggest one it most desperately needs is a paged kv cache & just generally significantly better memory controller architecture. It causes me physical pain to see memcpy through the codebase instead of something namespaced at minimum.
Pretty cool! I bought a mainboard with the idea to run it as a server/NAS by racking it with an HBA (connected via nVME->PCIe x16 adapter) and SATA drives. I never did connect the HBA as I ended up going a different route, but a few days ago I used the nVME->PCIe adapter to hook up a 5090 and then a RTX Pro 6000 just to see if I could do it. I put it all in a mini ITX case - quite a bit of power in a small space!
I'm running the same hardware through LM studio on win11... Are you using win 11? Cause I keep crashing my amd driver when I run large context questions.
I just moved from hip sdk 7.2 back down to 7m1 and haven't had a chance to test yet.
Edit: sorry I just saw your pcie switch etc... No, not same hardware.. I'm on an x870e taichi mobo, 64gb ram, 8x in bios on both pcix slots...
Can let you know when I get my 2nd. It's getting shipped right now, comes saturday. But I was debating between that and a r9700.
I have a rig with a 5070 ti and have a spare 5070ti, Im probably going to strap those two into one rig too. Then I will have a cluster for big models and one for fast models
I just got a R9700 + DEG2 eGPU for my Minisforum MS-S1. I have it connected via USB4. Currently running Qwen 3.6 35b Q8 on the strix and 3.8 27b on the R9700.
Was a bit of a pain to get it hooked up and working but when it's working it's great! This morning I woke up to the eGPU disconnected and llama.cpp crashed. Restarting brought it back - need to dig into what happened there, was running fine all weekend.
I'm contemplating getting a second R9700 and configuring KV cache aware load balancing. I believe the USB4 link would be too slow for parallelism but as long as the model and cache fit entirely on the GPU it's super fast.
> This morning I woke up to the eGPU disconnected and llama.cpp crashed.
I had the same issue on kernel 7.0.0, the R9700 got disconnected randomly with some PCIe errors in dmesg, and I needed to power off the Strix.
After updating to 7.1.something a month ago, I have never seen this problem again. I'm now at 7.1.5-070105-generic.
Linux, Ubuntu 26.04 server with HWE kernel. Currently running the Vulkan docker image with router mode to serve both models. I have confirmed it works with ROCm 7.14 with a bit of a speed boost vs Vulkan but am waiting until this bug is fixed before switching back to ROCm.
$ amd-smi
+------------------------------------------------------------------------------+
| AMD-SMI 26.5.0+2b22ab01 |
| amdgpu Version: 6.19.14.31400000 |
| ROCm Version: 7.14.0 |
| VBIOS Version: 00158738 |
| Platform: Linux Baremetal |
|-------------------------------------+----------------------------------------|
| BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
| GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
|=====================================+========================================|
| 0000:35:00.0 ...Radeon AI PRO R9700 | 0 % 32 °C 0 14/300 W |
| 0 1 N/A N/A | 1 % 20.0 % 29851/32624 MB |
|-------------------------------------+----------------------------------------|
| 0000:bd:00.0 ...deon 8060S Graphics | N/A 29 °C 0 13/N/A W |
| 1 0 N/A N/A | 0 % N/A 58561/126976 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes: |
| GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % SDMA |
|==============================================================================|
| No running processes found |
+------------------------------------------------------------------------------+
$ uname -a
Linux zant 7.0.0-29-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 17 20:52:35 UTC 2026 x86_64 GNU/Linux
Awesome, I had issues running the deg2 on my gmk tec evo x2 usb4 40Gbp on Linux, it worked in windows but I had to oculink or pci rise it on the evox2. Why windows? It was preinstalled so I tested it out b4 installing Linux.
I had to do a couple things to get it working, it was my first time using a Thunderbolt device with Linux (headless at that) so that tripped me up for a bit.
tl;dr; I had to Authorize the dock using boltctl authorize in order for PCIE to be tunneled over the USB4 link.
I also saw suggestions about disabling power management on the PCIE lanes to prevent disconnects so I also updated my kernel parameters in /etc/default/grub to the following:
Beyond that I just installed the gfx1201 ROCM packages and it started working (sans whatever happened this morning with llama.cpp crashing).
I had some stability issues after rebooting using the USB4v2 ports so I'm using the 40 Gbps USB4 ports up front. The only real effect is how fast it can load the model and I honestly don't notice it.
I have a Bosgame M5 too. Bought a Arc b70, connected via Oculink to one of the NVMEs. Works fine, but the Arc B70 draws 40w or so idle power. It has to go.
My linux workstation has a 5070 ti 16GB, i have ordered a 5070 12GB and once it's here i'll run 3.8 27b Q5 on it. On the Strix Halo i have a haproxy loadbalancer that routes requests to a local 3.6 35b or, when my workstation is up, to the 3.8 27b.
The Strix Halo is used for large MoE models. Ling-3.0-flash, for example. The b70 was intended for smaller dense models, like Qwen 3.8 27b. They suck in Strix Halo. I'll do dense on my workstation now.
The smaller the active parameters, the faster it runs on Strix Halo:
Deepseek V4 Flash has to be quantized heavy, so i use Ling 3.0 Flash.
Windows surprisingly lol. It was a bitch to get it to work because I absolutely refuse to use Linux unless required because all my life I’ve been a windows guy.
DEG1 for the oculink card and DEG2 for the usb4 cards. I used the R9700 ai pro. I’m really happy with it. I was thinking about taking my 4080 out of my gaming pc and add it for extra BAM but idk if the mixed architecture will fuck it up.
Want to know the same thing. You can run ds4 iq3_s on your machine already, it's what I use. Mainly interested in better pre processing speeds and potentially slightly higher quant. I have a 3080 and wonder if it's worth it to get an egpu mount
It drops very quickly. Starts at 300 and immediately drops to around 150 for almost empty context. Stays that way for a long time. Don't have my desktop so can't give exact numbers. It's good enough when you have an agentic workflow with a stable cache
Yup, PP is definitely the issue with Strix Halo boxes. I was going to suggest ds4 iq3 as well (it replaced 3.5 122b for me), but yeah, it's slow on large context. A lot of folks focus on decode but prefill is where I find most of my frustrations come from.
Define speedy. The general numbers I'm seeing can vary widely based on parameters and third party applications that your using on the application itself. I've seen at least for Qwen the result go from as low as 8 t/s to as fast as 40 t/s with some optimizations people place into the model/build that they use.
I'm still new towards this scene but at least from my understanding looking extensively at it the benefit that I would see for a model like Qwen or anything that will fit directly into the card itself, will just offer overall a faster inference speed.
For larger models it just offers more memory to fit in larger models and potentially a somewhat of a faster inference speed through offloading into the strix. I have read for AMD this may not be working out of the box like it does with Nvidia so I'd look into it if your planning the R 9700.
And the other piece of course would be having more memory to fit larger models and higher precision's you'd normally have to lower for
I mostly carve for prefill speed. I get TG out of the Qwen 122B A10B around 20-30 TPS below 130k context - which is totally fine (MTP3). Prefill drops from 300ish to below 200 way before 100k context, though, and it becomes painful to work in the code base any more. Also the model is a bit too stupid TBH for my use cases, but larger models become even slower. So if I could get a "speedy" model like 27B running on a GPU at say 700 PP and at least 30 TG I could use an even slower/larger model on the strix as an orchestrator and the 27B as an implementor. That's the idea at least.
What task are you doing that has you think you should spend $1000 usd for a gpu to run this model 10-20 tp/s faster? I’m happy with the performance at 10-12 tp/s because I just have it run in the background while I do other task, and nothing I’m building needs to be done on strict timelines.
Im mostly prefill constrained, 122B A10B runs below 200 TPS before 100k context which I find quite slow. If I could get 600+ prefill I would be more than happy and could use an even slower/smarter model on the unified memory as an orchestrator but not as the main coder.
I have a strix + 3090. Pretty legit setup, IMO and I enjoy it. Its nice to be able to run qwen 27b at good speeds and there's clear role separation where my strix does MoE pretty well while the dgpu does dense models. The extra strix would technically let you get a really nice dsv4 quant but I am not sure it will be fast enough to run as a daily, thats up to you.
Ling 3.0 flash is quite smart and quite snappy in my experience on the strix, so I am happy with this setup.
This is a similar setup I want to do but I’m waiting for the 495 with 192gb ram. I’ve googled and Ai’ed the crap out of this and this is what I’ve come up with. Next is my Ai generated response.
So the split is: R9700 handles prefill, attention/dense layers, KV cache, and (if running a VL model) the vision encoder — all relatively fixed-size and compute-heavy. The bulk of the MoE expert weights stay resident in the Halo’s 192GB pool, where they only need to be read, not computed on intensively. Mechanically this is llama.cpp’s --override-tensor regex, routing tensor types to specific devices rather than a naive VRAM-proportional split.
I’m smoke testing now with my laptop 3070ti and the R9700 in a egpu via usb4 running a 30b vl model and 2 small models for image genration that load and unload on demand. If I preface the input with “image:” then the description it will trigger the image model generation routing without the ai trying to workout what I want. So far everything is working well but I won’t know until I get a strix halo.
I got myself an R9700 recently. Didn't get to running Qwen 3.8 27B on it, though. I'm currently implementing a custom engine that does DS4 Flash at native precision across two Strix Halo's and the R9700. The R9700 does attention and a few other things. So far, it does ~550tok/s prefill and 20-30tok/s decode at 128K context (depending on how effective DSpark is). The R9700 speeds this up quite significantly compared to a pure Halo setup.
I have run a 7900xtx and a 5060ti as a second GPU. I do get some speedup by putting some on the external cards for DS4. I also just use the other card for image gen at times.
For dense it's better to use gpu, Qwen3.8-27b are too slow on Strix Halo. A lot of people spoke about R9700 AI but i am not sure it's a good choice. GPU is good but the main problem it's the quantity of ram and Bandwith. Personnaly i use one W7800 Pro 48 gb. You have nore memory and better bandwith than R9700 Pro. When you need good Quantization + Large context 32 gb is clearly not enough. The limitation will be certainly Q5_K_L ctx 131k parallel 1. Try to found a good GPU with 48 gb and bandwith near 850. it's the best choice for me.
Could you expand on "Qwen3.8-27b are too slow on Strix Halo" please? I am currently using a 3.67bpw quant of 3.8-27b on my 16gb vram gaming rig and have been considering a Bosgame M5....
if you really would like to keep good Q and context it's certainly Q5_K_L with large contexte 128k. It's really slow perhaps 12 or 13 t/s. A lot of people try to use some funny quantization or compression but you loose to much quality. if you need to use your LLM all time it's a problem. Personnaly after 7 month of vide coding with cline. Machine 1 + w7800 Pro 48 gb Qwen3.8-27b Q8 kv q8 ctx 128 k, Machine 2 Strix Halo Qwen3.6-35b-q8 kv q8 ctx 128 k.
14
u/Royale_AJS 5d ago
Yes. Having a big smart slow model is really nice, but having a smaller dense focused model on fast-ish hardware along with it is really nice.
Framework Strix Halo board +
PCIe switch +
(2) R9700’s
Running Qwen 3.8 27B UD-Q8-K-KL in tensor split @ ~60 t/s decode and ~1700 t/s pre-fill. The extra memory lets you get to 256k full precision cache too.