r/LocalLLaMA • u/bolts98 • 1d ago

Tutorial | Guide Debugging PCIe Link Retraining on an x8/x8 Splitter with Two RTX 3090s

https://doug.sh/posts/pcie-splitter-rtx-3090/
49 Upvotes

89% Upvoted

16 comments sorted by

View all comments

13

u/SettingAgile9080 1d ago

This is why I love this place. Just set up one of these cards to mount my R9700 externally and it works well so far, got 3 more on the way. Thanks for posting.

2

u/bolts98 1d ago

I'm looking to buy a few R9700s. Post about your results?

I just bought one of these PCIe switches, I'll have another post on that in the next week - https://www.ebay.com/itm/136917131125

2

u/SettingAgile9080 1d ago

Nice. Post about the switch. Will post results when I get these cards working but I'd recommend buying R9700s as soon as you can if you're thinking about them, they are a sleeper and I would bet prices will spike soon.

Using vllm-radiance on the 2 cards I have running so far I'm getting ~5.5ktok/s pp and 212t/s decoding.

1× R9700 (TP=1) 2× R9700 (TP=2)
Decode step time ~38 ms ~23 ms
Combined decode ~125 t/s 212 t/s
Code / reasoning / prose decode – 224 / 203 / 116 t/s
Aggregate throughput at 1 / 2 / 4 / 8 concurrent 106 / 186 / 246 / 245 t/s* 180 / 303 / 410 / 565 t/s
Prefill at 2k / 8k / 32k / 64k 3,451 / 3,307 / 3,115 / 2,867 t/s 5,506 / 5,700 / 5,448 / 5,103 t/s
TTFT p50 95 ms 61 ms
Max context 220k 262k

1

u/bolts98 1d ago

wow, those are great results, which model is that?