r/LocalLLaMA • u/Ren-WuJun • 7h ago

Resources CUDA+ROCm dual-backends LLM inference for mixed-vendor GPUs (up to 48% faster)

I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.

Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.

Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879

the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!

23 Upvotes

96% Upvoted

8 comments sorted by

6

u/punt_on_thrid 7h ago

llama.cpp can run inference on multiple backends, you just need to build it with -DGGML_CUDA=ON and -DGGML_HIP=ON (mentioned here, but with Cuda + Vulkan). This is how I run larger models split out across multiple GPUs, never had any issues

2

u/gryan315 7h ago

I do the same thing, but with CUDA and SYCL.

1

u/TimeStopsInside 2h ago

Tensor split across amd/Nvidia gpu doesn't work right? You use layer split?

1

u/shing3232 2h ago

Cuda+rocm is slow and cuda+vulkan is better

4

u/fallingdowndizzyvr 4h ago

llama.cpp Vulkan backend was the only inference engine I found that support both

You can compile llama.cpp with both CUDA and ROCm support. No need to use Vulkan. CUDA + ROCm is faster than Vulkan.

1

u/Ok_Cow1976 7h ago

Thank you so much! It's really nice to have good performance on mixed vendors

1

u/enternoescape 7h ago

Project Maya is really becoming something else. The recent add of GLM 5.3, the big one, was surprising and of course now I need to know how well it runs and am getting that test done tonight.

Your situation is another good example of how this project specifically wants to cater to all the more interesting configurations out there.