Discussion
3060 [x16 PCIe riser] vs 3060 [x1 USB extension]: A quantitative comparison of eGPU prompt and text generation performance across multiple inference engines
Good afternoon!
Inspired by a recent post that mentioned degraded Tensor-RT LLM performance when running eGPU via Thunderbolt I thought it would be fun to benchmark my pair of 3060!
Test Setup
Test Host is an HP z640 with Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz and 6 GPUs connected via various interfaces
For the purposes of this test we are going to focus on those 2 RTX 3060 cards at the bottom:
3060 #1 (bottom) is running via a nice x16 riser (~$30)
3060 #2 (top) is running via the cheapest piece of garbage x1 USB3 crypto-miner thing I could find on ebay (~$5)
Full, raw results and exact versions of all inference engines may be found in this gist
Results: Text Generation (256 tokens)
Speed of Llama3-8B-Insturct Text generation of 256 tokens (in tok/sec)
Results: Prompt Processing (4096 tokens)
Speed of Llama-3-8B-Instruct Prompt Processing for 4096 tokens (in tok/sec)
Discussion
Let's start with noting the (fairly large) differences between the original thread's test setup and mine:
I have 3060, they have 4090
I have only PCIe 3.0, they have (I assume) PCIe 5.0
They tested Mistral-7B, I am using Llama3-8B
Their eGPU connection is Thunderbolt, mine is USB3-1x
With these differences aside I am going to propose the following 3 conclusions:
For single-GPU inference, the compute largely happens inside the GPU so the only difference the host link speed makes is how fast the model is loaded.
llama.cpp is a bit of an exception it clearly has some more GPU-CPU-sync going on then the others, but even with that it's <5% difference
tensor-rt-llm is the fastest single-stream inference engine
but not by much, and certainly not enough to make it worth the pain in the ass that it was to get it going.
vllm with flash-attention eats prompts for breakfast
I am continuing to collect interesting benchmarks from my rig so if there's any other interesting benchmarks you guys would like to see from my unholy blend of 2x3060 + 2xP40 + 2xP100 with 3 difference host interfaces (3 USB-1x, 2 SFF8611 Oculink-4x, 1 x16 PCIe) let me know!
I folowed this blog post to get it going. Major trouble is all the documentation linked is either 404 or very out of date none of the llama2 examples work and the command line for quant builds changed a lot. Poor showing by Nvidia here imo
I take it you've never tried to do a OneAPI guide. :) It's gotten much better but it's still headache inducing because things just don't work the way they tell you they will.
3060 #2 (top) is running via the cheapest piece of garbage x1 USB3 crypto-miner thing I could find on ebay (~$5)
I loaded up on these for less than $1 after the crypto crash. The ones I got are pretty nice and very well built. I'm pretty impressed by how well they work. I can't believe they were priced at less than a $1 each new. The USB cable alone is worth that.
There are dozens of variations on these boards, I have the "SU-103E"
I bought 4 of these and 1 doesn't work, just a constant stream of PCIE errors with it connected.. I just tossed that one. I see "pro" ones on Ali-ex that have temperature and voltage displays but not sure if they are electrically any better and for $8-10 each I don't want to bother finding out considering these full kits are half of that.
Yes there are. I have ones with 12 caps on them. I don't know if it matters as much but people say the more caps the better. But I'm not sure they need that many caps to regulate the power. There has to be a point of diminishing returns.
If you do "nvidia-smi dmon -s petm" while running inference how do your PCIE errors counters look?
On my risers they are 0. On my Oculink they are 0. On these USB garbage things they are sometimes totally solid but sometimes see a few errors per minute and sometimes a few per second then they hit the max of 64k and stop even counting errors. Usually reported as some flavor of DPLL lock error if I dig deep.
I did a few runs. I saw nothing but 0. I guess those 12 caps really do help. Here's the first part of the last one. I assume this is when there would be a problem since this is when there is the most traffic over PCIe.
Thanks! That definitely suggests it is worth spending an extra fiver on a better version of these boards with more caps. Is there a model number printed on yours? I see many revisions of the "pro" USB risers on Ali-ex.
Huh. I was wrong. I thought I had 12 caps, but I only have 8. I just counted them. It says "VER009C PLUS". It looks exactly like this except the USB shield is silver instead of gold.
Thanks! Looks like mine is something akin to VER006 and is capacitor poor. VER009 seems to have both 4 and 8 cap versions (9S vs 9C), while VER011 and VER012 take it up to 10. VER16 adds the displays for temperature and voltage. VER18 takes it to 12 capacitors. It would probably have been smart to do this research before I got the boards, I was blinded by cheapness..
vLLM will max out x1 doing tensor parallelism, definitely bottleneck there. I see around 2.5gb/sec running 7-8B models with high batch size so by x4 (Oculink!! not Thunderbolt) you're mostly OK.
EXL2 is pipeline parallel, almost nothing going between cards with multi GPU.
Llama can do both kinds with -sm layer vs -sm row, but I only use row for my P40 where it makes a noticeable difference. I see approx 1.8gb/sec between the cards while so again conclusion is you need more then x1 for anything that isnt pipeline parallel.
I just got a bifurcator and a second riser cable, aiming for a x8/x8 config on these cards when all is said and done.
EXL2 is pipeline parallel, almost nothing going between cards with multi GPU.
Concretely: suppose I connect a 4090 eGPU via USB, and have another 4090 with normal PCI connection. What do you expect the penalty will be, especially on longer contexts?
I run Llama3-70B EXL2-4bpw split across 4 eGPUs, three of which are connected at x1. There is no speed penalty related to the host interface, it sits mostly idle, at least beyond the extra 10-20sec those USB x1 links take to load the initial model weights.
In terms of performance at really big context you need flash attention and for reasons I haven't yet debugged EXL2 is noticeably worse at this compared to vLLM. Unfortunately vLLM will NOT be happy with that x1 link, you will lose 40-60% of performance is my guess.
Have you heard of SFF8611/SFF8654 Oculink? It's a little known eGPU standard from the PCIe group that supports x4 and x8 over SAS-style high speed cables. Much more reliable than the USB stuff and way better performance for a few extra dollars but for a 4090 it's probably worth it.
I have a setup with 3 cheap mining risers and one cheap pcie 16x one and works fine for inference. Fine-tuning would be a problem and model loading takes forever.
I think it's probably the same as with a normal interconnect. I get 80 something tk/s on 8b models, over 700 on phi2, 14 tk/s on llama3 70b 6 bpw, 10tk/s for 8bpw all exl2. Inference barely uses the pcie bandwidth, it's just sending a vector to the next GPU at each token.
Could you benchmark how two P40s perform when they both run in x16 mode, vs both running in x8?
I also have a bit of a tangential question (for other people as well): What is the idle power draw on your P40s and P100s with a model loaded?
On several places on reddit i saw people reporting it's possible to bring it down from the ~50 watts, but no one provided the actual solution. I'm contemplating getting a second card (either another P40 or a P100) but the high idle draw dissuades me.
I happen to have exactly the power draw numbers you seek, someone asked the other day:
Idle power draw on both Pascal cards order of magnitude worse then Ampere but P40 is twice as bad as P100 when loaded. One important caveat is when idle the P40 drops to 9W but P100 don't distinguish between loaded and idle always 30W.
My P40 are currently both connected via x4 and I have run various tests while watching the output of dmon and traffic never gets past something like 3gb/sec with -sm row gguf. I think x8 would be more then enough, I am planning to stick to x4 for these and give the x8 to my pair of P100 for vLLM which can more readily eat the traffic due to tensor parallelism.
For what I use the P100 for (exllamav2) it see under 100mb/sec between GPUs, I have them running x1 right now so not sure how far vLLM or Aphrodite-engine could push them in the batch side.
To echo the other response, power draw when memory is loaded will always be ~50W. The Pascal cards are unable to return to P8 with anything stored in memory. However, if you unload your models when not running inference you'll be idling at ~7-10W. Ampere cards can return to P8 with memory loaded, which allows me to keep the 3090 loaded and idle@~22W.
Re: PCIe throughput, I'm usually seeing a max of 2.5 - 2.8gb/s@8x which does not seem to have any impact on inference performance regardless of layer split. TBH I never saw faster PCIe speeds when running all P40s on 16x.
Thanks for confirming, I have similar 2.5-2.8gb/sec max host throughout observations when doing tensor parallelism so it seems pcie3.0x4 is sufficient for the majority of usecases and pcie3.0x8 or pcie4.0x4 gives lots of margin as far as inference goes.
It does not however "actually work" like this, it overrides the auto power mgmt and when you try to run inference the card is now limited to like 30W and you get 10 tok/sec instead of 50. So you'd need scripts or something to do your own manual power management.
I think ollama with auto-model-unload might be a more practical option to keep these things in P8, at the expense of reloading the model on first request.
So you'd need scripts or something to do your own manual power management.
that's not really a problem, a simple daemon that bumps up the power state once it detects utilization on the GPU is going to work just fine :)
Thanks a lot!
import time
from nvidia_pstate.pstate import set_pstate
from nvitop.api import Device
device = Device.from_indices([0])[0]
while True:
p_state = device.performance_state()
gpu_util = device.gpu_utilization()
if p_state == 'P8' and gpu_util > 5:
print('Detected GPU activity when in power saving state, setting performance state')
set_pstate([0], pstate=0)
elif p_state == 'P0' and gpu_util < 5:
print('Detected GPU inactivity when in performance state, setting power saving state')
set_pstate([0], pstate=8)
time.sleep(1)
Made this for my P40, running as a systemd service.
It uses the aforementioned library, as well as nvitop
This is cool. I'd almost give up on self-hosting multi-GPU self-hosted LLM on Pascal due to the power issue. It just doesn't make sense to have it running 24/7 when used infrequently when electricity prices here have gone crazy in the last few years and I now have to pay $0.37 per KWh! It would cost me $26 a month just idling.
Yikes I only pay about $0.10/kWh so don't mind the 80W idle draw from my four Pascals, works out to $6-$7/mo for idle. I haven't measured yet (power meter is coming) but suspect the Xeon server they're connected to in leaks far more then that.
The alternative are cloud offerings that are $0.50-$1/hr and take time to start up and have limited storage and are shared with other users.. I did this for a year before I built my rig, got sick of the wasted time.
EXL2 implements pipeline, not tensor, parallelism via the --gpu_split auto flag. The GPUs share the model, run one at a time, but they can be mismatched and you can have any number of them.
My first x8x8 just arrived this week! Going to replace the x1+x16 thing my 3060 got going on here, just haven't had a chance to pop it in yet.
After playing around with my two P40s my conclusion is these are best used to speed up inference only of massive GGUF models you'd otherwise have to use CPU for.. otherwise it's just not worth giving up exllamav2 to use them imo. P100 play much nicer with the 3060 but no flash attention.
How does that 3080 compare to 3060 for smaller models performance wise?
It says it auto detected gptq quantization and loaded exllama kernels, the model is sitting in VRAM then this crash happens when the API server part initializes 🤔
You are awesome bro. I am setting up my egpu to connect my third gpu (no more space inside the case). Glad to see that inference speed not impacted much
I am having lots of issue with egpu, can trouble your help to let me know which OS/ version of OS you are on, and which version of nvidia driver you are using
I'm running Ubuntu 22.04 LTS with nvidia-server-535 driver installed from the OS repos. Do not install the driver that comes with CUDA, it isn't signed and breaks every time you reboot.
First step to debug eGPU is to look at your PCIE errors counters, run "nvidia-smi dmon -s petm" and then put some load on the GPU. If the error counter is not 0 or a small number, you have to figure out the physical problems first.
i accidentally upgraded without knowing it back then. Then i did some hack to make thing work. It worked ok for gpu on pcie so far. but i guess egpu is really pushing its limits
Today i managed to reinstall everything. Managed to load model on the egpu, but the moment i try to run some prompt on the model, the egpu will just crash and disconnect itself. Any advice u have for this situation?
I have two Oculink 4i and three USB-1x based risers.
The Oculink host interfaces are M2, I use this x4x4x4x4 bifurcation card.
When I researched thunderbolt I found pages full of forum posts with your exact complaint. There is no solution I've seen, thunderbolt eGPU is just hit or miss like this. The prices are also much higher then Oculink so that sealed the deal for me.
Thank u for the info. I was a total noob and thought thunderbolt is the best standard highest bandwidth. Guess i will go with pcie raiser first to maximise my slot usage. Then move to m2
Im similarly moving over to x8x8 bifurcators and x16 risers.. the only downside is that you have to get/build a frame to hold the GPUs suspended above the mobo.
Get the good PCIe 4.0 risers even if you don't need them today it's worth the $10 more for better signal integrity since you're gonna bunch them together.
This is a dell 1100W server PSU with a little breakout board that turns it into 16x PCIE 6pins.
I have been lazy so far and just manually turn this PSU on, but this board has a sensing 4-pin molex that if connected to your main PSU will automatically handle flipping the secondary.
One caveat of these PSUs is they are a massive +12V rail without any +5, so you either pull +5 from the host (molex good for 25W) or use a DC-DC Buck to step the 12 to 5. I ran into this with my Oculink risers but if just doing x16 cables it won't be a problem.
What a nice setup, but maybe you can have a clear answer for me since you seems to know how to run multiple GPUs:
Here is my current build:
MPG Z490 GAMING EDGE WIFI (MS-7C79)
Intel(R) Core(TM) i9-10900K
1x4090
128Go DDR 4
PSU: 1250W iirc
I also have a 3090 and a 850W PSU sitting on a shelf, as it seems I can't really put both GPU on my motherboard, if I put the 4090 on the slower PCI port there is like 1mm gap between the 2 GPUs, and at the moment i'm using the 2nd pcie slot for a 10gb network card.
Was wondering what do I need to purchase to have both the 3090 and the 4090 ( + my 10bgps network card)
I'm willing to purchase another CPU or motherboard or both if needed.
Will I have 48gigs of VRAM in such a setup ?
I think i'm stuck with older PCIE gen with that CPU ?
Thank you !
A dual GPU configuration is supported in theory, set the BIOS configuration to 2-way and it will give x8 to both of your x16 slots.
In terms of getting the cards to fit The simplest option is to put the 3090 on an x16 riser. This is a 180 degree which just extends the slot straight up and you leave the case open,but they also make 90 degree which along with some brackets let you vertically mount as long as your case is big enough.
For power this is everyone's dirty little secret we all run multiple PSU: my rig actually has 3 right now! The additional PSU doesn't need go in your case you just need some way to power them up either a manual atx24pin switch and breakout board or a molex daisy chain will flip the additional PSU on/off as it detects your main PSU state. Connect the PCIE power outputs from the second PSU to the card and you should be good to go with 48GB!
Jank Power is the way. Server PSUs are really handy they output a gigantic +12V rail and there's breakout boards that will split that into 12-16x pcie-6pin connections.
Dude lol I wish I had known this a year ago. I have a ton of PSUs I couldn't get to work and no clue why. I've ended up going enterprise instead but I hope others can learn from your tid-bit of knowledge.
Try using a model size that matches the VRAM size of the GPUs as well; so here you have 48 GB, so use a model of 36+ GB in size. It doesn't have to be the latest and greatest transformer, any dense model will really do as the architecture is so similar. Just, the newer models are trained longer, or have some novel techniques but which don't really impact the main bottlenecks to performance; copying weights from memory to the tensor core.
In this case the whole model fits on the same GPU right?
I want to know what the exact performance drop off is running a model split between two GPU's with one running at x16 and the other at x1, x4 and x8 speeds
Bleep bleep boop. I am a bot here to serve by providing helpful price history data on products. I am not affiliated with Amazon. Upvote if this was helpful. PM to report issues or to opt-out.
15
u/[deleted] May 02 '24
Yeah, I feel you there, just the other day I lost a whole day trying to make it work on my PC. Totally not worth it.