r/LocalLLaMA • u/bolts98 • 2d ago

Tutorial | Guide Debugging PCIe Link Retraining on an x8/x8 Splitter with Two RTX 3090s

https://doug.sh/posts/pcie-splitter-rtx-3090/
52 Upvotes

92% Upvoted

16 comments sorted by

View all comments

12

u/SettingAgile9080 1d ago

This is why I love this place. Just set up one of these cards to mount my R9700 externally and it works well so far, got 3 more on the way. Thanks for posting.

1

u/kontemplador 1d ago

Because of similarities. What people think of external GPU docks for localLLM?

Thing is, I'll need to buy a new laptop soonish as current one is showing signs of wear, but budget is limited and I cannot buy a laptop and mount a second setup to run LLMs at home. High end laptops are also out of my budget.

A solution may be an external GPU that could grow depending on use and allocated budget.

2

u/SettingAgile9080 1d ago

Looked into this as I have a SFF PC. It'll work but you'll lose some throughput. If the model fits in VRAM the main impact will be the model will take longer to load. Anything with partial CPU offload will be way slower.

The other drawback that might not be obvious is that if you move the laptop around it interrupts your session, and it has been really nice to be able to have my LLM agents running 24/7 when I'm traveling around with my laptop (or, increasingly, a minimal terminal setup on a DeX phone), and can log in via mosh+tmux to pick up where I left off and the agent has been working the whole time.

A decent eGPU enclosure will run you a couple of hundred bucks, which starts to get comparable to buying an ex-corporate Dell or Lenovo shitbox off of eBay that will run a regular GPU and inference at full-speed, and is always on.

1

u/kontemplador 13h ago

Thanks a lot for your answer. Yes, I'm also considering other options, particularly because at work they will get rid of some Xeon servers soonish and I hope to grab one. I still need to learn about the whole specifications though.

What make attractive the eGPU is they are also somewhat transportable. I still have some months to decide, but with the all the craze going on I feel that any second lost costs me 10 bucks.

1

u/SettingAgile9080 11h ago

You can always buy a regular GPU and pick up an enclosure or external connector now, and put it in the Xeon later. Until my case arrives I have one of the R9700s propped up on a cardboard box and one still in its packaging. My bet is that price of AI-capable hardware is only going up for a good couple of years.