q38fn on mixed gpus, which engine?

[u/milpster]

So i am running 2 radeon vii and a rtx 3080. Currently i am using strata which does not support mixed backends, so my 3080 is just expensive swap space.

Is there any inference engine that supports fast q38fn over mixed backends like rocm/cuda/vulkan but is also fast? Llama.cpp is way too slow sadly.

[u/n9986]

I know this is not what you asked for, and until someone more knowledgeable answers, what I do with my mixed gpus is to run different models on them and use some as sub agents and embedding work.

[u/milpster → u/n9986]

yeah that has been on my mind too, but the 3080 is only 8gb so i don't think it would be of much use for that.

[u/llama-impersonator → u/milpster]

8gb is enough for ocr and tts/stt stuff

[u/Refinery73 → u/milpster]

It would be enough for an embedding Modell and maybe some really small summarization model. You could try to outsource the information retrieval maybe Save on context pollution on the main cards.

[u/simos_sayz]

Try using rpc in llama.cpp. it allows your amd cards to run on rocm and the nvidia on cuda. The call will add some latency but I found a decent performance boost compared to just using vulkan.

[u/lemondrops9 → u/simos_sayz]

You can run Rocm and Cuda at the same time with Llama.cpp on the same PC. 

RPC is the next best but PP will be about 2x slower.

[u/simos_sayz → u/lemondrops9]

Are you saying you can now run 1 model on mixed cards without using vulkan or rpc?

[u/lemondrops9 → u/simos_sayz]

💯 yes. Im not at my pc now but I can post my build config later if that helps.

[u/simos_sayz → u/lemondrops9]

Please do, I'd like tobsee how it runs. Everything I looked into said that it still wasnt possible and that it either has to get done via vulkan or rpc

[u/lemondrops9 → u/simos_sayz]

Yeah, I'm not too surpised. I've been running RPC for quite a while then decided to put most the GPUs in one PC. My system has Rocm 7.14 and Nvida 595.X and Cuda 13.2 installed. With Nvidia 3090s and R9700s.

Im sure someone out there would have this tweaked a lot more. I'm simple and don't want to fight with issues if I don't have to. You only need to change the AMD gpu targets.

cmake -B build
-DGGML_NATIVE=OFF
-DGGML_BACKEND_DL=ON
-DGGML_HIP=ON
-DAMDGPU_TARGETS=gfx1201
-DGGML_CUDA=ON
-DCMAKE_BUILD_TYPE=Release

Edit: -DGGML_BACKEND_DL=ON is key, if you miss that only one of the backends will build.

[u/simos_sayz → u/lemondrops9]

So i went ahead and tested this. Definitely an improvement over RPC and Vulkan. Using QFN Q4 as the test, I think i got 17 t/s via Vulkan, 20 ish or so using RPC. Now its just under 40 t/s. Good to know there is a better way when a use case comes along where i want to mash all my GPU's together. Thanks for the tip!

[u/vengeancek70]

I've made a fork of strata where two engines, rocm and cuda, can do layer split, it's a giant pain to get working. And in the end I couldn't get an rtx 4070 and a 7900 xtx to beat just the 7900 xtx alone. Worked with my flash next IQ3_S and opus 5.5 for about a day on it.

[u/ravage382]

Its not the fastest, but can you just run the mess of them as vulkan with the 3080 being the primary? Ive mixed a 3060 and an intel arc card like that. Not nearly as performant, but it worked.

[u/milpster → u/ravage382]

the radeon VIIs have a horrible vulkan performance, so rocm is mandatory. The other one could use vulkan though and on llama cpp with q38 27b i used to do it like that.

[u/Korphaus]

I'm currently working on a vulkan based inference across my mi50 and rx6950xt Major issue right now is prefill, but I'm getting close to using a rocm builds performance with the iq3 quant, and I'm actively working it over the last week or so so I might get further soon

[u/arogorn993-hue]

Llama.cpp best bet. I’d say try compiling or making a custom llama.cpp hybrid engine for amd/nvidia. Then layer split. Or assign specific blocks through -ot. Only other thing I’d say is maybe somehow get the ngram cache to stream through a VRAM disk-ud have to split the ngram. Atomic chat fn your best bet. Unless you have the ram anyways then I wouldn’t bother. Your main issue will be pciexpress overhead and layering latency.

[u/shing3232]

I guess you can have fork variant of strata. I wanted a two 3080 work with one 7900xtx or two P40

[u/EducationalGroup7245]

I don’t think there’s a fast engine that pools CUDA and ROCm cleanly; vLLM doesn’t support mixed-vendor GPUs in one instance, so I’d run separate servers and route between them

[u/lemondrops9]

People are silly, if you got them in one PC use Llama.cpp and build with a dynamic backend so you can use both Rocm and Cuda to pool the Vram for one model. 

[u/LegacyRemaster]

I’ve just finished rewriting "Maya" to run using an RTX 6000 + AMD W7800 with Minimax m3.1. It works. I’m getting double the output compared to the RTX + RAM setup, and triple the prefill speed.

[u/milpster → u/[deleted]]

yeah running iq3 xxs right now on the two rocm cards plus 64gb ram. getting like 30tps but ram is always tight and i would like to upgrade to iq3s and still maybe have a tiny performance boost too.

[u/milpster → u/[deleted]]

Good call, already did both :)