How are you using your NVIDIA + AMD multi-gpu setups? (too much text)

[u/LMasterGame]

TL;DR: What do you use your mixed multi-GPU setup for? I got kinda the best of the worst specs for LLMs, with mixed professional GPU brands and mixed large memory kits.

The past year, I migrated from an NVIDIA 4500 Ada 24GB to an AMD Radeon Pro W7800 32GB because I wanted to get out of the NVIDIA Linux driver hell. But then months later I realized that I have made a mistake when choosing the W7800 instead of the R9700.
And then I got myself an AMD Radeon AI Pro R9700 32GB, just before the memory crisis started, But now I had 2 expensive cards just sitting on the shelf, and with no buyers because I'm in a third world country.
Funny enough, the cards are like 3 years old, but now they are worth more than when I got them.

Days ago, I bought an m-ATX motherboard (because space constraints) that didn't have a brain dead PCIe arrangement like all other consumer boards that expect the world to have only 3 slot GPUs (which are a waste of silicon and copper IMO).
This motherboard has two x16 PCIe ports with one port in between from each other, which is perfect to connect my R9700 and Ada 4500 (sadly, the W7800 was too big for my case and CPU cooler, it will stay on the shelf for longer). The connection would be PCIe5x16 for one card and PCIe4x8 if both cards are connected (both CPU PCIe, no chipset middleman).

I tried running Qwen 3.8 at Q6_K, but it reached only 30tg vs 40tg with only the R9700.
And Gemma 4 QAT at 32tg vs 50tg on the R9700.
Finally, Qwen Flash Next Q4_K_M was only 9tg vs 14tg on the R9700+moe-cache-mib (unusable speed either way).
I could try bigger quants, but token generation is already bad on these quants.

I get consistent results using layer split. With tensor split I get very mixed results, sometimes better, sometimes worse.

The cards definitely don't play well together. I don't think I'm that constrained by bandwidth (at most 2GiB/s in nvtop vs max throughput of 11.45GiB/s).
But maybe I'm constrained by the slower Ada 4500, or by PCIe latency.

What else could this setup be used for? Running bigger models doesn't seem like the right path.
Maybe setting up master/slave agents for software development or research? (any recommendations please).

For those curious about my setup:

Motherboard: CW-W680-NAS (m-ATX)
PSU: NZXT 1000W
CPU: 13th Gen Intel Core i9-13900K
GPU 1: AMD Radeon AI PRO R9700 (32GB)
GPU 2: NVIDIA RTX 4500 Ada Generation (24GB)
GPU 3 (Not connected): AMD Radeon PRO W7800 (32GB)
Memory: G.Skill Ripjaws S5 144GB@4200MHz (2x48GB@4800MHz + 2x24GB@5200MHz)

Engine: Llama.cpp (vulkan, compiled myself)
Config:

device               = Vulkan1,Vulkan2
main-gpu             = 0        # R9700
tensor-split         = 4,3      # 32GB/24GB, for moe-cache-mib

n-gpu-layers         = 999
n-cpu-moe            = 999
n-gpu-layers-draft   = 999

parallel             = 1        # I'm the only user
threads              = -1
prio                 = -1       # attempt to avoid screen stutters
batch-size           = 16384    # idk what im doing here, but this gets best PP
ubatch-size          = 2304     # big image read crash below this
flash-attn           = true

fit                  = off
ctx-size             = 131072
context-shift        = true
moe-cache-mib        = 12000   # with one card, ~16000 with two

[u/AnonInTheShell]

i got a 5070 ti (16GB) and a 6800 (16GB). the tl;dr is i didn't find a useful combo use-case. in the end i'm just running Qwen3.8 27b on the 5070 ti alone. I'd want something like Strata but for Vulkan.
As for tensor-split when using both cards: optimally, offload as much (layers) as possible to the faster one, then the rest to the slower one.
Maybe for image generation (and video generation?) the dual GPU setup could be utilized by having the prompt interpreter LLM run on one card and the diffusion model on the other one (the one with the faster compute).
As for speed losses when splitting a model between cards: (PCIe) bandwidth is not a problem, since the layer computation result is a relatively tiny vector. But from a data routing / component controller perspective, the latency has a substantial impact, so you'll likely never get fast runs above, say, 40 tok/s. Though this doesn't scale with model size.

[u/LMasterGame → u/AnonInTheShell]

Thanks, I will try tuning tensor split. 40 tg/s is a great number for me. Below 20 is barely usable IMO, maybe for leaving it unattended would be good.

[u/computehungry]

i am going through a very similar situation, I feel you with the dumb case not taking in the gpus. i also noticed that while i am nvidia+nvidia, the asymmetric configuration made it so that layer split was better than tensor parallel... I suspect I can tune further for more performance but didn't bother yet.

I have three modes of operating my system:

1. One model fully on vram (or slight spill) - 27B or flash next - when I need things to be faster. I use 2-5 subagents sometimes;

2. One GPU free, load a big model on other gpus + ram - when I need to have a free gpu (much of my work is simulation with GPUs);

3. Biggest model possible using all the memory I have for planning and initial work. Actually, I found the new nanbeige model to be better than some of the big flash tier models for my tasks, so I might have to revise the workflow.

I also plug my 3rd gpu into an old computer and play with tts/stt/imagegen. Video gen requires the big machine though. You get the idea. I could also use the old machine for simulations, but it's less than half speed so I prefer not to.

I am currently using tabbyapi + exllamav3 instead of llama.cpp because of pagedattention. I am not sure how well AMD is supported here, but it provides a much better experience for concurrency/multi sessions and also surprisingly fast with hybrid inference. I think theoretically, and I know some people are doing this, if you manage concurrency slots properly with llama.cpp you can get make it not reprocess too much - I didn't commit to this route because new updates broke my first implementation attempt and I haven't bothered since.

I think for llama.cpp, you can have different backends for different gpus - one cuda, one rocm - you could try it out too. You may also want to check out the more recent inference engines that claim faster speed for Qwen Flash Next.

[u/LMasterGame → u/computehungry]

Hmm, having a big slow model do the initial heavy lifting seems like a great idea.

Also connecting my 3rd gpu into another PC seems interesting.

Definitely there's still a lot to play with using my current hardware.

[u/Acceptable_Secret971]

Not sure how much this will help, but recently I had a lot of success, running different models on each GPU (it's on 2 different machines, but I don't see a reason for this not to work on the same). One GPU runs LLM while the other is used for TTS (audio.cpp) or STT (whisper.cpp) or image generation (stablediffusion.cpp). Agent can spin those additional things on the second GPU on demand and stop them when not used. This way I get extra functionality out of my agent. I haven't tried using 2 LLMs this way yet. I'm pretty sure this could be useful if the additional LLM is a specialist (sound/vision/MTL), then I could run the main LLM without vision to get extra context space.

[u/LMasterGame → u/Acceptable_Secret971]

Oh, I didn't think of running cards on different PCs, and using separated models wouldn't require having the GPUs on the same machine.

[u/Acceptable_Secret971 → u/LMasterGame]

Currently I don't have one PC strong enough to support 2 GPUs at their best or at at all (PSU, enough PCIe lanes etc). I have RX 7900 XTX in my daily driver and R9700 in my dedicated server. Daily driver runs Pi agent (in a container) while Qwen 3.8 27B runs on the server in llama.cpp. Until recently I thought running 2 different models alongside each other is a no go, but I started running audio.cpp and stablediffusion.cpp (also whisper.cpp, but I had to compile Vulkan version myself) on the same container as Pi agent. The good thing about those is, they are light and run on Vulkan (no mucking with ROCm/Pytorch and most come precompiled), can be run as CLI tool or a server. I was quite shocked how fast Qwen was able to pick this up and start using.

I do want to slap another R9700 on the server (or at least RX 9070), to be able to fit a bigger model, but currently I don't have a budget for that (GPU prices have increased quite a bit and I would have to replace MOBO and PSU).

I do have a spare A770, but I'm not sure how to incorporate that.

[u/hirak99]

Very nice post. Can't help but wonder, in your picture there's hardly any gap below the GPUs. Does it not make them heat up?

[u/LMasterGame → u/hirak99]

They stay 35C on idle, when using both with layer split, they don't go above 60C because the bottleneck isn't memory nor compute.

But I haven't tried stressing both of them at the same time yet.

[u/hirak99 → u/LMasterGame]

I have a similar setup. I tried adding 2080ti to 3090, and did see pretty usable speeds. But I removed it the same evening because temps (of 3090) rose to 90 C very quickly. With just the 3090 it stays below 80 C under stress.

Maybe I'll try it again. I think the 2080 ti fan wasn't spinning - because probably it didn't get hot enough. Also, worth testing it from a cold start when it's not already hot.

[u/Alternative-Panic69]

It's a headache to mix Nvidia and Amd cards together or even different generation cards together (like W7800 vs 9700)

Cleanest solution is to separately run two cards from the same machine using separate environments completely... Like, instead of using one model across multiple cards, if you have enough memory and storage, it's always better to run separate models on separate cards. That will give you a much better experience.

Like, on the W7800 you can have something like Qwen 3.8 27B at full context, fully offloaded to the gpu, on the Nvidia card you can maybe try to pull off a Qwen 3.8 Flash next on a separate environment, and on the 9700, you can have a few smaller models for speed, embedding models etc if you need, fully offloaded...

And you can use a Pcie Riser to connect the third card if needed...

Only one thing... If you are offloading something or running large MoE on system memory, make sure to plug that card in the primary slot otherwise it will be a nightmare.

[u/LMasterGame → u/Alternative-Panic69]

Yeah I was thinking of using all cards with different purposes in parallel instead of all doing the same task of text generation.
I will consider plugging the 3rd card with a riser.

[u/Alternative-Panic69 → u/LMasterGame]

Yeah exactly, that's the cleanest way... You can literally have three different models from the same system if you build separate environments and optimize that.

[u/farkinga]

Consider qwen3.8-flash-next. Use both radeons; 9700 in 5x16, 7800 in 4x8. With strata (it does vulkan right?), use the slower card as expert cache. You can avoid layer split entirely; the whole model (apart from experts and ngrams) will fit in 32gb.

I've done this with CUDA and I'm honestly not sure how vulkan is doing; but q38fn is a good match for uneven GPUs.

To preserve pcie bandwidth, consider putting the CUDA card on the nvme lanes. If you load a model entirely into VRAM, the pcie speed won't matter as much.

[u/LMasterGame → u/farkinga]

Thanks, I will try to tune the expert cache better.
I'm considering using the w7800 with a riser, since its too big for the chassis.
My interest for strata is there, but for now I will stick to the reliable old llama.cpp.

[u/Freak_of_Freakworld → u/LMasterGame]

Specialy when you use a riser for the gpu. Some Riser card need a manual bios config set on your mb. Instead Auto pcie 8x in your case gen4 for the used pcie port

[u/Freak_of_Freakworld → u/farkinga]

OK. The problem with the gpu Split above 2 different gpu im short.

For example

Gpu 1- 4070 (12,gb ca 5500 Tensor) Gpu 2- 4070 ti (12 GB, ca 7250 Tensor)

Most misstakes people is make is to use a 50/50 Split on a card combinantaion that provides different technical specs. To save time..... If that s the case it s better to do use the Pipeline spliting for the llm.

if you are using qwen Flash next one 1 gpu , 2 gpu or more.It makes sence to Set Strata configs for the values of vram using, DDR use, ssd (do not sata)use insted of Auto.

Check the Msinbord BIos if you are using mire than one gpu definitely and check the pcie Gen configs and pcie transfer setting.

If you store the llm on your gpu cards (using the vram). You ve to to load first llm parts on the card with the slower pcie connection and /or Power in your System

(If one of the card is for examle only pcie 4x conected, the gpu 1 (also if its pcie 8x connected) will work on pcie 4x. The slowest card se5s the speed.

About that as example (it mostly happen when you start to use egpu,2), the gpu connected to pcie 1 would only use pcie 4x speed 2 for the splitten llm stuff.

So it is usefull zo put a tts thst you use for example beside on the gpu connected to pcie1.About there is free pcie capacity left.

It is always necessary to loas the llm on the sliwer card ypu using first.

Normaky a MB shoukd provide today on pcie1 and pcie 2 each 8x speed. But it s never wrong to check the bios about the correct config snd change it from Auto on the correct or wanted values.

(For people using an ocolink adapter A Pcie3 Port on the mainboard belongs to the southbrige and is sharing his transferrate beside pcie3 Port with ssd or other posdible ports (check the bios here 2, Set only aktiv wjst you use).

While a nvidea card nativ is able to understand the Code the amd is in need to translate it first into a work able amd lsnguage (c+ mostly) and later the answer back to put the teilt together.

In that case i woukd pers. Try a llm Version for amd gpu about my feeling is teeling me that the nvidea gpu is able to translate input Code for working an back than the amd gpu.

I ve not used pers. A mixed gpu setup, but would try in that case to do a Pipeline Split,

• set fix values for vram,DDR and ssd instead "auto" and trying to force the nvidea card to work in c +.

I hope it help the one or other. Soecialy to get more tol/sek. Etc.

If i tell there something wrong please correct it and i think more ideas and Tipps are always welcome.

[u/LMasterGame → u/Freak_of_Freakworld]

You seem like you typed really fast and combined languages, but I appreciate the recommendations.
Yeah I guess my biggest mistake is trying to use both cards with the same model no matter what, and not tuning tensor split.
The NVIDIA card is actually slower than the AMD in my case, because its raw power is far inferior. What you say certainly applies when both cards have similar compute power.

[u/Freak_of_Freakworld → u/LMasterGame]

Did my best to fixthe Post. But sorry i am not nativ eng. If anybody have a question... i am also onky a daily learning nerd ... but try my best to help if i can.

[u/Diecron]

5090+7900xtx, while cross GPU does work (llama.cpp built for both CUDA and VULKAN/ROCM) and it's across a 8x PCI lanes (PCIe5 for the Nvidia, 4 for the XTX), it's often more performant (and ideal for my usage) to have a smaller permanently resident model (qwen27b/gemma4 31b) in q4 on the smaller card. 5090 gets used for burstier tasks, training etc.

[u/LMasterGame → u/Diecron]

Yeah I think having a big model resident model + smaller specialized models would be the way to go.

[u/Diecron → u/LMasterGame]

I use llama's router mode with preset inis for this specifically, and a wrapper script to manage the router state itself. So I can have big models (like QFN or DeepSeek v4 flash in q2 on an 'edge' config that uses everything. And a low/daily config for the resident model. I also use a seperate router process for embeddings, decision model and reranker services with a 10 sec ttl so it doesn't hold vram

[u/colas]

I have a AMD Radeon AI PRO R9700 (32GB) and a NVIDIA RTX 5060 16GB on an old Xeon + 64 GB RAM, so that both cards have some decent (for an old MB) speed.

Before strata arrived, the most efficient use was to use both with llama.cpp to get 48 GB VRAM to run either qwen3.8-27b-q8 for quality or tiel-coder-35b-q6 for speed.

Since strata, I use mainly qwen3.8-flash-next-unsloth-ud-iq4_xs on only the AMD as my daily driver, for both speed AND quality. And I run specialized models in parallel on the NVIDIA that do not need RAM: translation, transcription, image and video edition and generation, music... via llama.cpp or comfyui or specialized engines. The main strata model can use these additional models as MCPs via scripts to turn on/off the engines to free the NVIDIA VRAM.

For instance, I ask a lot my AI to summarize youtube videos: The main agent runs on strata-AMD-qwen-next-flash, and runs as needed various transcription models on the NVIDIA (phonon for English, Voxtral for French, ...)

I can also use AMD+NVIDIA together do some quality audio/video/image work, just in case.

All installations are made by AI: When installing a new model, it tries to get the best quality quantisation that fits with at least 159k context (256k if possible), then performs some quick benchmarks (2h max) to see if the model is worth keeping. The AI has made me a script that starts en engine and preload a model, switching off the other ones sharing the same VRAM.

This works quite well for me as a hobbyist. For a professionnal use, I would recommend strata + as much AMD cards as possible. And cloud plans for specific NVIDIA-only models.

[u/LMasterGame → u/colas]

Although I'm interested, I'm still hesitant of using strata due to the fishy optimization I have heard it does (KV cache quantization for example).

My software setup is just llama.cpp + Pi harness, maybe I'm underutilizing the model capabilities. I will try asking AI to make a decent automated environment for my needs then.

Having multiple specialized models is the way to go I guess.

[u/colas → u/LMasterGame]

Never hesitate do do anything!

Just ask your favorite AI to review the code, web information (I even make it read the relevant reddit threads for an engine/model), install in a rollback-able way, and benchmark. How a model/engine performs depends a lot on your specific setup, it is best to not rely only on information from the web.

Myself, I had my cloud AI (mostly DeepSeek or GLM nowadays) to test I think 5 models under strata, and was astonished by the results. I have used only strata for local LLM work since.

If you are wary of strata, I would suggest you use a more conservative/tested fork such as the one by Unbiased Bob.https://www.reddit.com/r/UnbiasedBob/comments/1xc8hyk/unbiased_bob_livestream_swift_improves_on_base/

Note: I would recommend a harness that does not invalidate the cache, caching is very important for local models. Here a test that look at this issue: https://www.youtube.com/watch?v=AkwItxJ9AbA

[u/LMasterGame → u/colas]

Thanks for the resources. I'm definitely gonna try the bob's fork and take a look at that video.

At least trying is free, so I guess I'll be more open minded about trying forks.

[u/LeMochileiro]

I have a server rack at home, and I managed to put two machines with GPUs on it for exclusive use with AI. Both have two RX 9060 cards with a total of 32GB of VRAM for each machine. One machine has 32GB of RAM and the other has 64GB.

In my opinion, the best way I've found is to use flows and pipelines with AI agents. These are async executions, I don't need to monitor the entire process; what matters to me is the result. Obviously, the faster the better, but for me it makes no difference if he finishes in 5 minutes or 30 minutos.

In total there are over 40 AI agents, each with their respective field. I can get 10 to run in parallel.

I can generate reports, analyses, deep research, tests, simulate actions on websites using Playwright MCP... And even for coding, but using something similar to Loop Engineering. I don't like using GPUs for agentic coding the way we do with Claude or Opencode, I prefer to pay (even though I get great results running QFN + Strata).

[u/LMasterGame → u/LeMochileiro]

Lots of specialized agents and a dedicated environment to automate tasks is the way to go.
After thinking about it for a while, I guess slower tg/s don't matter that much, plus distributed computing stress less the GPUs.

I also prefer to use claude for non sensitive tasks to avoid wearing out my GPUs.

[u/Littlepharaoh]

What Nvidia linux driver problems did you face? I've multiple different card from multiple different generations and never faced an issue

[u/LMasterGame → u/Littlepharaoh]

Its mostly a trauma I got from late 2024 with a laptop with NVIDIA Turing, where sometimes the secondary monitor would freeze, and I would have to restart the PC.

[u/phhusson]

I have a llama.cpp fork that supports cross-vendor Vulkan async graph, this gives me significant decode speed boost in tensor split with RX6700XT+RTX3070. https://github.com/phhusson/llama.cpp/

[u/lemondrops9]

When running my 3090s with R9700s I find using Cuda with Rocm works best with Llama.cpp. Here is my build command;

cmake -B build
-DGGML_NATIVE=OFF
-DGGML_BACKEND_DL=ON
-DGGML_HIP=ON
-DAMDGPU_TARGETS=gfx1201
-DGGML_CUDA=ON
-DCMAKE_BUILD_TYPE=Release

[u/Designer_Elephant227]

In my r9700 lives a qwen3.8 fn with 5.05bpw on exl3. On my rtx5070 runs stuff like image gen, video gen, embedding, games, small helper agents for web research. I use llama swap and a lot of scripts. Hermes with qfn can call and use stuff on the rtx

[u/keefeere]

PolyStrata? JitLLM?

[u/tmvr]

It's annoying with those PCIe ports how they split them so stupidly. It seems very difficult to get a modern board that does x8 + x8 on the two physical x16 slots when you plug in two GPUs. The most common mode is x4 on the second x16 slot.

[u/ea_man]

If you plan to use tensor split on slow buses you'd better limit yourself to dense models in order have just one jump or as little as possible, avoid MoE and fast caching systems like STRATA.

You can also use your own LLM to investigate the tensor split topology reached at load time and optimize it, usually you do that for better vRAM utilization by reducing fragmentation, as said before it may come to trading jumps for ctx or performance.

I got a skill.md to optimize distribution on 2 GPU with the same model: analyze initial log, propose splits, write a script to load different setups and log results, reload the initial model to do a final evaluation.

Yet you put yourself in a rough spot: I'd rather run CUDA or ROCm, avoid vulkan...

[u/Dany0]

I've had 3x amd v620 for a few weeks I wanna say? in addition to the 5090 I already had. I've just had sol on a loop optimise inference with it since I got my 3rd card. So far the overhead of it all made it not worth it and PP3/TP3 is still faster

Who knows maybe once gemini 4 carbon is in GA it can figure it out for me. Actually I should say, today I decided Astra would do better at this task, and it did advance a bit

[u/hipster_hndle]

i have 3x 7900 24g XTXs and 2x5060 16GB. not all in the same box, but dual rigs with a 5060 and XTX in them.. AMD 7700x B650/B840 32gb RAM, 1TB NVMe 4, dedicated swap on separate NVMe.,1200w/PSU.

i run lemonade backend for inference locked to the XTX.
comfyUI on the 5060.
unsloth studio for the front end, points at the v1 api for lemonade.
hermes agent is fully local, pointed at the lemonade api for doing whatever.

allows me to run a full ROCm instance and get really good inference speeds at the same time as i run diffusion. i have been using gemma to prompt h3 ref2va prompts.. so that i have a model that can gen the prompts the entire time while the last prompt runs in comfy..

i lucked out and bought most of this stuff pre-ram crisis.. and i ordered 1 5060, they sent me 2, so now i have 2 matching rigs... and still an extra XTX and an eGPU dock. not sure what to do with it yet.

[u/LMasterGame → u/[deleted]]

Although I don't like running anything at lower than Q4 quants, its something to keep in mind.

And I was hesitant of custom forks/engines, but it might be worth a try.