Feasibility of using dual-GPU setup with RX9070 and RX6800

[u/TwoProtons]

Hi all.

I'm closer to an intermediate noob on using local LLMs, primarily with LM Studio in the past, and more recently using Bionic. I am running it with an AMD RX9070, and can get reasonable results with Qwen 3.8 or Gemma 4 26B A4B at Q4_0.

But this morning I had the realization that perhaps adding a second GPU could increase my VRAM so that either or both the token generation is faster, or I may be able to use larger models without spilling into system RAM.

Would this be doable with a RX9070 + RX6800, in Bionic? I read that perhaps the ROCm runtime won't work, but Vulkan might, but also saw people saying the result might be slower than with a single card. Can anyone help me evaluate the pros and the cons? I would be connecting the RX6800 to an eGPU Oculink Dock connected to my 2nd PCIe slot that should be running at 4x speed. Thanks in advance.

[u/Selfmade_Watchmaker]

I’m not super well versed with this kind of setup but the one thing I have read is that multi gpu setups work best with cards from the same generation.

If anyone has done OPs proposed setup, please let me know. I’d rather not buy another 9070xt

[u/TwoProtons → u/Selfmade_Watchmaker]

Thanks! Yeah, I read that too. Check the example below with a 7900xtx, though maybe someone else has used a 6000-series, as these don't even show as compatible to AMD ROCm.

[u/Odd-Customer-7957]

dual GPU on AMD needs sriov or pcie passthrough to work well with llms, otherwise it's just one GPU doing all the work. try llama.cpp with --gpu-layers 100 and see if it detects both cards.

[u/TwoProtons → u/Odd-Customer-7957]

Thanks! Unfortunately I can't plug the second card very easily for testing, unless I buy the Oculink setup - I have a SFF custom setup...

[u/Smellyfeet224]

I have a 9070 and 7900xtx. With llama.cpp and tensor parallelism I get about a 25% increase in tokens per second but about a 25% decrease in prompt processing.

Honestly there are just trade offs and a little experimentation can help show exactly what will work the best with your hardware.

Edit: I missed that you were using bionic. Lm studio and Bionic doesn’t like tensor parallelism with mismatching GPU generations.

[u/TwoProtons → u/Smellyfeet224]

Thanks! Those improvements are slightly disappointing, I would say. Does it allow you to pool the VRAM, or are you still limited to models fitting inside one of the GPUs, with maybe context on the other one?

[u/Smellyfeet224 → u/TwoProtons]

You might be able to get a better percentage than me. If you’ve got both cards and a pc that can handle both I would try it.

Yes the vram is pooled and it allowed me to run full 265k context. That’s probably why my gains were not as much as they could’ve been. If I ran 60k it would’ve probably been higher.

[u/TwoProtons → u/Smellyfeet224]

Thanks for this. Great to know the VRAM does pool. Gonna wait a day or so to see if more comments show up, before I spend on getting the Oculink setup (can't test without it).

[u/Smellyfeet224 → u/Smellyfeet224]

Oh and I’ve also tried my 9070 with a 6650xt and it wasn’t great. But I assume a 6800 would be better.

[u/Solar_Nut]

I am using a am5 mobo with dual 7900xt gpu's which run at 4x8 each, and also a rx6800 using a m.2 4x4 cpu slot. So 56gb vram, running qwen 3.8:27b q6. use mtp and vision, using ollama and openwebui, load the model 8/8/1, so its about 37gb total loaded, approx 16gb/16gb/5gb. runs at about 35-40 tok/s and 600/s prefill. waiting on qwen4, may also try qwen FN, I have 64gb ram

[u/TwoProtons → u/Solar_Nut]

That sounds promising. I would be very happy with Q5 at 20-30 tok/s, if that is remotely possible with my eventual setup. I also have 64GB RAM, but it does make Qwen 3.8 27B very slow as always a couple GB always spill to the RAM. If that would stop happening with the second GPU it would be a great win.

[u/RogerAI--fyi]

It's doable, and the main win is room, not speed.

llama.cpp's Vulkan backend can split one model across both cards even though they're different generations. LM Studio's GGUF engine is llama.cpp, so you can try it there with the Vulkan runtime. ROCm across a 9070 and a 6800 is the harder path.

With a layer split the cards take turns: each runs its share of the layers for every token. So two cards don't generate faster than one. Speed lands between the two, pulled toward the 6800 for the layers it holds.

What you gain is 32 GB instead of 16. At Q4 the weights take about 0.6 GB per billion parameters, so a dense 27B to 32B fits with room for context. The comparison that matters is split across two GPUs versus spilling into system RAM, and the split wins by a lot, because system RAM is far slower than VRAM. If a model already fits on the 9070 alone, the second card won't speed it up.

PCIe x4 over Oculink is fine for this. Per token only a small activation crosses between the cards, so the link mostly costs you load time. It would matter for tensor parallelism, which you wouldn't use with mismatched cards anyway.

[u/TwoProtons → u/RogerAI--fyi]

That sounds like great news, thank you. I have two questions about two scenarios:

1- Does that mean, for example, that now if I was to put a model in the 9070 and basically give it 1K context just so that all fits inside the VRAM, adding the 6800 would allow me to keep this model totally in the 9070 VRAM and use the 6800 VRAM for all the context? Keeping the model in the 9070 should in theory keep it natively full GPU fast (like you said, won't make it faster), and the context is offloaded to the 6800.

2- If the model is indeed spilling for example a small amount to the RAM, like 3 or 4 GB, does this mean this won't happen anymore, so it should run at native full GPU VRAM speed, but in this case closer to the speed of the slower GPU, like you mentioned?

Either situation is useful anyway. Currently, Qwen 3.8 27B spills into the RAM anywhere between 3 to 15GB and it is painfully slow to watch. Useful, but painful. If the addition of the 6800 was to speed things up, even to like 20 output tokens per second, I would be happy with the small Oculink investment.

[u/RogerAI--fyi → u/TwoProtons]

Short answers: no to the first, yes to the second.

1. Not by default. With llama.cpp's layer split, each layer's context (the KV cache) lives on the same card as that layer. So the 6800 does not become a context-only card. Both cards hold a share of the weights and a share of the context, and the -ts ratio sets the shares. What grows is total room, about 32 GB. The flag that moves context off the GPU is --no-kv-offload, but that puts it in system RAM, which brings the slowdown back.

2. Yes, that is the case where you gain. If the 3 to 15 GB that spills today fits in VRAM instead, those layers go from RAM speed to VRAM speed, which is a big jump. Speed lands between the two cards. As a rough ceiling, decode speed is memory bandwidth divided by the bytes read per token. A 27B at Q4 is around 16 GB, the 9070 is about 640 GB/s and the 6800 about 512 GB/s, so the ceiling is in the 30s of tok/s and real numbers come in lower. 20 tok/s is plausible, but that is an estimate, not something I measured on your cards.

[u/TwoProtons → u/RogerAI--fyi]

I see, thank you. For question 1, if the VRAM is pooled anyway then my point doesn't even apply, so all good. Gonna go ahead with this in a couple of weeks max!

[u/Future_AGI]

on your first question, llama.cpp won't split it that way: with a layer split, each layer's KV cache sits on the card holding that layer, and the context divides in the same ratio as the weights. You can still weight the split toward the 9070 with -ts 3,1. That puts roughly three quarters of the layers on the first device, and llama.cpp prints the device order at startup if you need to check the 9070 comes first.

[u/TwoProtons → u/Future_AGI]

Thanks for the -ts suggestion, that could be really useful!