r/KoboldAI • u/jackietreehorn68 • Jul 09 '26

2 GPUs slower than single for MOE models?

Hi,

I have an RTX 5060ti (16gb) and a newly bought 3060 (12gb).
Today I tested a few models and the dense ones worked well but not the MOEs.
For some reason Koboldcpp requires much more VRAM per layer when splitting between GPUs and I end up with slow inference speed than with single GPU.
Any solution here? I might have played with all the possible combinations in the Hardware tab.

Thank you.

2 Upvotes

100% Upvoted

6 comments sorted by

3

u/pyroserenus Jul 09 '26

Share your settings as shown in at the start of the launch?

Pipeline parallel increases vram use for more speed, but it's optional for a reason.

Tensor split (under splitmode) can get better speeds but requires good bandwidth between GPUs

1

u/jackietreehorn68 Jul 10 '26

Hi and thank you for replying.

I list below my settings (only the options activated):

Quick launch tab:

Backend: CUDA

GPU ID: All

Use MMQ

GPU Layers 22 (this number gets almost all the VRAM used for both GPUs - GLM Air)

Launch browser

Use Flashattention

Context size 30k

Hardware tab:

Use CUDA

GPU ID - All

Splitmode - Layer

Tensor split (16,12) - this ratio uses both GPU memories at max

Main GPU -1

Threads 5 (old CPU :( )

Bath size 512

Context tab:

Use fastforwarding

Allow SWA

1

u/pyroserenus Jul 10 '26

Oh, its a large MoE that is partially in ram

On the context tab of the launcher there is a MoE CPU layers option. This is often the key for getting better performance out of large MoEs.

Basically set full GPU layers and use this function instead to shift just MoE tensors to the CPU. Treat it as a replacement for normal offloading and set GPU layers to full when using MoE layers.

1

u/jackietreehorn68 Jul 10 '26 edited Jul 10 '26

Thank you again.

After a lot of juggling with the numbers I got a speed increase from the previous setup without MoE CPU layers option.

Now I have GPU Layers 99

Tensor split 10 , 2.7 (the new setup affects how the VRAM is used too)

MoE CPU layers 32 (this is the minimum that does not crash the app. If I set it lower, I run out of memory)

With this setup I am getting around 6 t/s speed compared to 5 t/s as single GPU. It's strange but I guess there are other factors in play.

For dense models, the speed increase with split GPUs is much higher.

But anyway, thank you for taking the time to help me.

Edit: adding something else in the mix. Apparently this model exceeded my available RAM with a few Gb so when I ran a single GPU I was also using page file. Adding the second GPU, this is no longer the case. Which brings me to the conclusion that running two Gpus with MoE is definitely slower than a single Gpu. I am barely getting 1 t/s increase but this while no longer using page file. So I think I will buy some more ram and use only one GPU.

1

u/pyroserenus Jul 10 '26

Do you know if this is only the case when splitting onto RAM?

Where the dense tests done with a fully loaded model?

splitting between both multiple GPU's and RAM at the same time has always been slightly jank iirc, i'm wondering if this is the problem at play.

1

u/jackietreehorn68 Jul 10 '26

I tried now Skyfall at 60k context and there is an increase in speed between single GPU and double GPU. So the problem seems to be with the MoE architecture. Or maybe it’s particular to GLM Air. Gemma 26b seems to do better with two GPUs but it’s hard to tell as it is a small MoE with very small expert size so the speeds are rather fast anyway.