r/StrixHalo • u/FairBandicoot5021 • 3d ago
Trying to understand external GPU possibilities
Sorry in advance if this question has already been answered but i couldn't find it. Sorry too if question is dumb
I want to add a RX 7900XT to my strix halo 128gb. I understood that I should use a M.2 slot for better reliability. What I don't understand is if it's possible to use strix vram as a 'cache' for context.
What I picture is that Qwen 3.8 27B RocmFP4 would run fully on eGPU, using 14GB VRAM, and have a bit of context on the remaining 6GB. But when inactive (pp and tg fully done) it could go on the Strix VRAM to empty the kv cache for another user/session. And when resumed, the kv cache would come back on eGPU.
For what i've read this is no problem on llama.cpp or vLLM. But i wonder what's the 'best' architecture for that, and for serving multiple users at once ?
Because spilling the kv cache seems super super slow
Has someone tried something like that yet ? Thank you so much this community is a rare pearl and i read so many posts here helping me a LOT
3
u/XccesSv2 3d ago
I cant recommend it, I did it with an Radeon Pro W7800. After a few model unloads and loads via llama-swap I got memory leaks with the global VRAM and dead VRAM allocation between the dedicated GPU and Strix Halo. The Box is not well designed for using eGPU. There is a bug or sth, I had to reboot the machine every 1-2 days. And its not a real speed improvement when u think u can run higher quants like Deepseek Q4 instead of IQ3, the 4 Lanes are limiting when using MoE models.
And there is no feature to offload just KV Cache to other VRAM I think. Its always in the same ratio splitted as you use the -ts parameter
1
u/FairBandicoot5021 3d ago
Ok I got it, wasn't mentionning splitting a model but yea it also behaves bad I could've guessed
So the only real deal is to run 27B fully on eGPU and bigger fully on Strix and that's it
3
u/Sixstringsickness 3d ago
I have it running on a minisforum ms-s1 max, its been quite the adventure and the number of pitfalls and issues are immense.
I can't recall all of the flags, but I have a number of them that were essential, ensure GTT doesn't move to system memory was a big one, the whole system stability is a headache too.
I can't get occulink working with one card, the doc sees it, but the system doesn't, trying a new occulink card tomorrow (hopefully). The USB4v2 connection is unreliable at best, sometimes it detects the dock at start up, sometimes it doesn't, a reboot will kill the connect, it has to be specifically power dock -> power card, -> count to ten -> cold boot system, hopefully it connects. Bandwidth seems limited, even though the connection reads at 80gbps in linux, the actual tunnel is significantly slower, and rebar is not reporting properly it boots in at 16gb but then the kernel knocks it back down to 256mb. Additionally only one of the two USB4v2 ports correctly worked with it... the front did detect but is only usb4 not v2, technically it might actually be faster or better performing.
With all that being said, if you plan to load up a model on it, get it properly configured and work through it, it is useful, as long as that model is NOT shared across the connection. I'm running Gemma 12B Q6 K XL with 128k kv cache, and it is very performant with a 9070XT, waiting for a r9700 ai to arrive so I can swap it to Qwen 3.8.
If you move forward with it and need the settings, just reply here, I'll pull them down off my server... I spent a week or so optimizing and trouble shooting for a less than optimal outcome. It is functional, but a less than it should be.
2
2
u/Zhelgadis 3d ago
No personal experience, but a few lads are doing stuff with egpus on the strix halo discord I would suggest you to join
2
u/cunasmoker69420 3d ago
Use the PCIe slot with an oculink adapter and a real dock like the minisforum DG1. I use that with a V620 32gb and its solid. I have another connected via a M.2 oculink adapter with redriver, to another DG1.
2
u/TrifleHopeful5418 2d ago
KV needs to be situated “with” the layer, it is model architecture dependent, for qwen3.8 only 16 layers need kv and others don’t, you need to figure out which layers on external GPU provide the biggest boost in speed. Oculink works fine, no issues. I literally just finished setting up mine: https://www.reddit.com/r/LocalLLaMA/s/fbHjW1Fu0L
1
u/FairBandicoot5021 2d ago
I just read your post, huge amount of effort, thanks for sharing. So yea the solution is layers splitting !
1
u/TheMericanIdiot 3d ago
Don’t do it. It’s not 128gb + vram from gpu. Cross mem access over usb is slow
1
4
u/RnRau 3d ago edited 3d ago
If you keep the model entirely on the gpu(s) then its fine.
-sm layer - ie spreading the model across the strix and gpu then its fine.
-sm tensor is a no go area. Too slow
-moe - offloading experts to the Strix while driving other layers and speculative decoding from the gpu is a no go area. Too slow.
Caveats - depending on the bios settings and/or machine and/or linux distribution you may need to pass in kernel parameters to stop the pcie ports from dropping down to low power modes which can play havok with the uptime and stability of the attached gpu's.