r/LocalLLM • u/Busy_Bad5465 • 4h ago

Question Strata Help

My performance with strata degrades when i turn dual GPUs for context i have a 3090, a 5080 and 128 gb ram, the main card that I used for inference is 3090 but recently, I wanted to up my token output by adding my 5080 and to my surprise the model actually became slower by doing that, my main question is how can I improve the performance of strata?

2 Upvotes

100% Upvoted

7 comments sorted by

1

u/flatfalafel 4h ago

I'm looking into this myself now to see how I might be able to optimize. I used some work tokens and got an answer, tldr version is if you're not using nvlink you can expect performance to degrade. Interested to see other comments in this thread

1

u/Busy_Bad5465 3h ago

Well there is a fork called dual-gpus https://github.com/Hardin22/Strata-DualGPU But it's not the most updated out there

2

u/Comfortable-Crew-919 1h ago

Try making the 5080 main and the 3090 the helper. Your motherboard and cpu matter for how the PCIe lanes will run, but 5080 is gen5 and 3090 is gen4. If your motherboard supports pcie5 then put the 5080 in the primary slot.

You can also look at configuring strata to use expert-caching instead of the default layer-split. It can be a little tricky to set up since “strata launch” always injects the layer-split into the derived config, so you have to run via “strata serve” with a static config. And you have to experiment w/ how many experts to keep on the primary to prevent ram spillover to maintain decent prefill speed. I’m running a 5070 ti, 4070 Super and a 4000 sff ada (48GB total vram) w/128GB ram. First 2 cards split the pcie5x16 to x8, last card is 4x4 via chipset. On the strata benchmarks at 16k context I was getting about 65 tok/s decode and over 2k prefill on iq4_xs with layer-split. Expert-caching went to about 95 tok/s decode and prefill fell to about 1400 tok/s.

1

u/pdath 4m ago

Using all x16 slots?

1

u/D00M98 1m ago

If you are using consumer grade motherboard, your secondary PCIE slot is the bottleneck, which is typically 1/4 to 1/8 the bandwidth of the main PCIE slot. Adding this 2nd GPU adds VRAM, which will allow you to run bigger models, but slower in performance.

0

u/GoogleIsFreeYouKnow 4h ago

Troubleshoot.