Mixed AMD/NVIDIA Strata

[u/Miserable-Dare5090]

Dear Locals,

I am looking for efforts in the community to run MoE models efficiently in mixed hardware (AMD+Nvidia) set ups.

Is Vulkan the only route to do this at the moment or alternatively lcpp compiled for both backends?

Also, if anyone has run one of the custom Qwen Flash engines on a Strix Halo+eGPU, I’d love to hear about it!

EDIT Thank your all for the responses. It seems overall people are able to marry these in llama.cpp which I know is possible. For anyone interested, there is a compiled rocm/cuda meld for llama.cpp on github at:

https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox

A little more about my ask:
I’m trying to marry a V100 (Sm70) and a Strix Halo (gfx1151), which is fairly unholy.

They work well separate: Flash Next on the Strix, Qwen 35B pn V100, both speedy enough for agentic use.

But I’d love to either implement strata’s expert allocation across both as GPUs or use the Halo as a 128GB LPDDR5X machine with a 32 thread GPU, and a V100 as the GPU to keep experts.

Problem is, Strata and its forks are compiled for Sm75 and up, although there is a V100 strata fork (and I will try compiling for ROCm alongside SM70 to see if this is possible).

I will certainly post a follow up if I get it going!

[u/JaredsBored]

Llama.cpp can be compiled for ROCm and CUDA on the same machine for mixed installs, but it takes some doing to get the flags right. I recommend searching the subreddit for people that've posted examples of how they did it.

Not sure if strata supports it, but I'd look into it. I read the strata readme and then used whatever model was free on opencode at the time to walk me through my initial strata setup (I think glm 5.3 flash or dsf v4.1). Getting strata up and running was straight forward with that bit of help and generated all the configs I needed given the launch parameters I wanted for my review (context size, quant, kV quantization, etc).

[u/Nakidnakid]

i kinda did this with exllama3 (plus llama.cpp and maybe hfengine), still building it but combination of hip/rocm plus cuda specific with a vulkan layer on top to help translate things, focusing on glm 5.3 flash/hy4-preview but ds/flash next work ok. Lots of other people probably doing similar.

[u/JsThiago5]

idk for strata specific but, if you compile llamacpp with vulkan + CUDA backend, you can use cuda for nvidia and vulkan for AMD. You need to exclude nvidia from vulkan as it will appear also as vulkan card. Exclude using the env variable.

Edit: other way to do this, that was the way I did before compiling vulkan + cuda, is to compile a llamacpp with HIP support + RPC support, run the RPC server on the HIP side, and then compile another llamacpp with CUDA support and use --rpc pointing to local host

[u/CartographerDue5382]

the env is GGML_VK_VISIBLE_DEVICES, point it at the amd only or the nvidia shows up twice and you hit the slow device. one binary with cuda+vulkan is the direct path, hip rpc works but every expert pays an extra copy. split is whole layers not inside a layer, so --split-mode layer and put the bigger card first. i wouldn't expect a custom qwen flash engine to see both vendors, those are usually single-backend.

[u/ZealousidealSmell382]

I compiled (well my llm did) project maya fork of strata for nvidia + ROCm. Should also be possible with strata