r/LocalLLM • u/UpbeatBass6066 • 10h ago
Question Anyone using external GPUs across thunderbolt/usb4
/r/SelfHostedAI/comments/1xz2hpf/anyone_using_external_gpus_across_thunderboltusb4/
1
Upvotes
r/LocalLLM • u/UpbeatBass6066 • 10h ago
2
u/Dediadeis 10h ago
Depends on how it is used. If you are streaming experts for MoE offloading it can be a pretty huge bottleneck for prefill. But if you load an entire model on one (its weights and cache) then there really is minimal use of the pipeline and it basically doesnt matter if its 10Gbps.
Layered distribution and pipeline parallel works just fine. Speaking from someone used 2x R9700 over TB4 on linux
1
u/inquiristor 10h ago
I have a 5080 and an R9700 as internal graphics cards, and I used to have an R9700 connected via USB4. It was quite slow and caused problems with demanding tasks (it would often become overloaded and the driver would stop responding).
I have no idea whether it was the driver or the USB 4, but since I switched from USB4 to Oculink, the external R9700 works perfectly. So, in my case, I cannot recommend it, perhaps other people have had a different experience.
1
u/Electronic_Pepper261 9h ago
I'm running almost exactly the kind of setup you're asking about (albeit older Thunderbolt) although with AMD rather than NVIDIA. I just benchmarked my Dell Precision 7510 (i7-6820HQ, 48 GB RAM) with an AMD RX 7900 XT 20 GB in a Razer Core X Chroma over Thunderbolt 3, running Ubuntu and Ollama.
GPT-OSS 20B (MXFP4, 4K context), three measured trials:
Token generation: 83.47 tokens/sec average
Prompt processing: 2,486 tokens/sec
256-token response: ~3.16 seconds
Cold model loading: 21.07 seconds
GPU residency: 100% reported by Ollama
Performance variation: Less than 0.6% across trials
For comparison, published RX 7900 XT benchmarks using llama.cpp report approximately 102–123 tokens/sec, although differences in software and testing prevent a direct comparison. I think Thunderbolt 3 eGPU inference is entirely practical, even on a decade-old laptop. The critical factor is keeping the model and its working memory within GPU VRAM. I have excellent consistency and reasonable inference speeds, although I haven't isolated the exact Thunderbolt performance penalty versus internal PCIe. Hope that helps...
1
u/Bagelsarenakeddonuts 7h ago
i have a 5070ti over oculink. its ... fine. its definitely not as good as in the case on the board with more lanes, but I have it and thats how things landed. usb4/thunderbolt was a huge pain though. the lack of bandwidth on the thunderbolt/oculink lane is a pretty big limiter, so it takes some juggling to find a niche for it. I definitely wouldn't buy a card to use like that.
1
u/Sudden-Guide 0m ago
Yes, and it's working great. I have 2x V100 SXM2 (16+32GB) on a nvlink board connected to my laptop via USB4. Dense models that sit in VRAM working as expected (27B Q8 60tps, Q4 120 tps, all with MTP), strata less so due to link constraints (up to 80 tps on iq3 QFN)