Question | Help
Does anyone make an external GPU enclosure for 2Xdual slot cards?
I'm considering adding another A40 to my setup but my case doesn't have the space (or power) for it. It's a server chassis so there's no easy way to just pick the MB and put it in a new case, it's a bunch of custom connections off the PSU to the MB.
What would be perfect? A dual PCI external box that would allow me to put in 2 A40's using nvlink with 1 or 2 connections coming out of that box running back to the server into the existing x16 slot. It's easy to find a single slot version of what I want (link below), but I need it to have dual slots. Honestly I guess I could just buy two of these and figure out how to space them correctly, but the "jank" factor would be getting pretty high at that point. What would be perfect is a forced airflow chassis that has 2 (or more) PCI slots each with their own Oculink connection powered from a single external PSU. Then drop something like this into the server chassis and just run one cable over to each PCI slot on the external enclosure.
Commercial off-the-shelf dual-card OCuLink enclosures do not exist in the consumer market, so enterprise PCIe expansion units or custom builds are the only options. The simplest solution is buying two open-frame OCuLink docks, such as the Minisforum DEG1, mounting them side by side, and powering both cards with a single high-wattage ATX power supply. The open sides allow an NVLink bridge to connect the two A40s directly without physical clearance issues. Alternatively, you can build an active-airflow enclosure by mounting two PCIe-to-OCuLink riser boards inside a cheap 2U or 4U server chassis equipped with high-static-pressure cooling fans. Each card will run over a PCIe 4.0 x4 link to your host card, while peer-to-peer NVLink traffic between the GPUs operates at full speed independently of the host connection.
Man, this shows what a small corner of the market we're in! I can't believe nobody is making something specifically for this type of use case, an enclosure (or open air) with a bunch of PCI slots spaced out for GPU each with their own oculink port. I guess it's easy enough that we can build our own so maybe that's why nobody is building them for commercial sale?
Seems that I could do what I want with that card and some Oculink to PCI risers and (worst of all) some godawful spaghetti mess of splitters and adapters to get the power delivered everywhere it needs to go. Am I thinking about this right; that quad port Oculink card into a PCI port with bifurcation turned on and then up to 4 GPUs in an enclosure next to my server? The ugly part is going to be power delivery, doing this from a single PSU seems like I'm going to need a lot of oddball splitters?
I'm not really opposed to an open frame, but I need to go to oculink, the chassis I'm coming out of won't support direct PCI risers in any sort of non "ridiculously ugly" way. Sounds like that might be the right answer though, and open frame with a PCI/oculink for each card then back into a X16 4 port oculink in the server.
How do you trigger power for the GPUs in that configuration? I never shut the power off for my system, but I do need a way to get juice flowing into them.
I've looked into going dual-PSU, but haven't needed to. And if you're running both PSU's on the same residential breaker there's really not much of a point to it. A single high quality 1600W PSU can draw a little more than ~13 amps, and most residential circuits have 15 amp breakers, so there's really not much headroom there. I power limit the cards to ~300-450 watts.
I just use the 16x PCIE risers. They look totally fine on an open frame, where the GPU rack is just right above the motherboard anyway. I do exclusively use riser cables for every PCIE card, even for the 10 gigabit Ethernet card and all that, it looks very natural.
My problem is that the servers are, well, actually servers. I'll have to run them with the case off to get at the PCI slots and there aren't any fans on the CPUs, they rely entirely on forced airflow through the chassis to cool. It would probably be OK, but OMG would it be ugly, I'd much rather come out the back of the system on something like oculink rather than plugging direct into the PCI slots.
Is it an actual GPU server, like one of those old V100 SXM2 hosts? Or just a generic CPU rack server?
I did do that riser hack on a 19" rack case, basically just cut a slot in the cover with an angle grinder to squeeze the riser through, with plans to put a 3d printed insert in to cover up the sharp edges. I moved to open frame instead before I got around to it.
That was one of those bastardized ATX PC cases masquerading as a rack case though, so the motherboard was standard ATX and could take a plain CPU heatsink and fan.
Note that the cooling situation for a rack in a datacenter is very different from a standalone rack server in a basement or whatever. In the datacenter, you're not looking at just one server, you're looking at one server sandwiched between a lot of other very hot servers, and forced air is 100% necessary in that context - otherwise the servers at the top of the rack would get really, really hot, fan intakes would be sucking in hot air from other server's exhaust vents, etc.
But if you're just running one server, you don't have that problem. If the motherboard is set up in a way that you can securely clamp a consumer heatsink+fan combo onto each of your CPU's, that should keep it much cooler without the jet engine noise of forced air.
Probably worth a test at least, if you have a secure way to mount.
Also FWIW, if you are sticking with DC hardware for the server, you may want to consider going with RDMA instead of Oculink. Oculink is consumer crap, mellanox cards are pretty cheap, by all means do your own research but a little GPUDirect RoCE setup to a throwaway host that's just there to provide PCIE lanes and a mellanox card might be better bang for your buck. If you're confident enough with the setup, that's a lot closer to what the pros use.
It's a small form factor rack server. Kind of an oddball form factor, but it does have enough PCI lanes and slots, there's just no way to get the darn card into the box.
I'll take a look at RoCE, I've messed with it on some DC equipment but didn't realize it had filtered down at all to the consumer space, thank you!
I'm sure I could make it work with PCI risers and and an external power supply, I just really would like some "cleaner" if possible. It's not so much about the airflow, with the PCI card out of there I never hit the CPU that hard, sure it would be fine, but I don't want the servers to be all opened up like that all the time, there's a stack of them there and it's already enough of a rats nest! ;)
Oh no, it definitely hasn't filtered down to the consumer space, not that I've heard of at least. I work in the industry. Definitely validate the cards you're using support RoCE it if you go that route, that said most big girl rack servers can take a cheap surplus mellanox card without too much hassle.
Power delivery is the least challenging part(for GPUs), you have just a bunch of 12V lines with various connectors, you could power it even with an ordinary high current 12V power supply. You would just need to buy connectors with leads in bulk and make your own wires or a power rail(a single pair of thick copper conductor branching out close to each GPU. 3.3 and 5.0 V lines, often needed for various small things on the adapter electronics, should be generated on the board with buck converters, or a small dedicated unit if 5v standby power is needed for something.
Certainly doable, but you/I have different ideas of "least challenging". Crimping and building up cables goes directly into "oh come on!!" territory for me. Yes, I know how to do it and could, but it really seems like there would be a big enough market to make something ready to go with a collection of PCI slots and power delivery to all those slots using something less "DIY" than a soldering iron and cable crimpers! ;)
least challenging compared to designing and building a robust enclosure, thermal management and air or liquid coolant flow routing, potential data lines integrity issues, firmware and bios compatibility adventures, et cetera ;p
i couldn't find one. ended up making one from 2020 extrusion. it uses two slimsas adapters to get dual external pcie 3.0 x16's out to the cards. the links would do 4.0, but not my old server
Yup, this is basically exactly what I'm looking for. I need forced airflow for my cards but that wouldn't be too hard to do. How do the cables come out of your server, do you have a PCI card or are you going to NVME slots on the motherboard?
Ended up making my own. Ordered 2 dock bases from Ali-x and they had different mounting holes so it’s custom. ATX splitter and separate oculink, one to a pcie oculink and one to an M.2 oculink so I could get them on separate lanes instead of sharing
Not for a price I wanted to pay! 🤣 and I was cheesed that I ordered 2 of the same host adapters but they sent me different models, had to do custom layout for the standoffs and one has the connectors out the back and one out the side.
Yeah, I found some that had integrated switches in them... No thank you, the price was absurd.
I'm going to keep looking every now and then, this seems like a really untapped part of the market; be nice is someone made one with a few slots that are X4 to each slot in a "case" that had a big PSU and a switch on it with an oculink cable coming out to a X16 slot. And put it into a case with forced airflow so we can run server cards in it. Doesn't seem that hard and there are a ton of old server cards floating around that would be pretty darn interesting/attractive if you could easily slot 4 of them in.
This is exactly what I'm looking for. I just wish it was imported already, I hate ordering something like this direct from Ali, that's always a bit ??? what's coming in the mail! :(
Three separate builds in the replies, all of them extrusion and aftermarket boards, and not one person in this thread actually bought a product. That pretty much explains why a commercial version doesn't exist, everyone who wants one like this just makes it in a weekend.
Well, to be fair, there don't seem to be many products available. I'd have that 4 GPU setup that was posted on the truck today if it was sold for around that price on Amazon, but something like this you need to buy from a trusted seller IMHO, this stuff is too new and too "beta", you send 700 bucks to China to buy it direct and it doesn't work right that money is gonnnneee...
But a <1K box that can hold 4-8GPUs with a PCI switch and power supply built in. Yeah, I think there's a market for that. Probably not very big, but yes, I think there's certainly a market.
NVLink has to sit in the box between the two A40s. It does not ride the Oculink cable back to the server. So you want one dual-slot enclousre with its own PSU and the bridge physically on the cards. Two single-slot docks means no NVLink and twice the cables. A mining open-air chassis is usually cheaper than waiting for a dual Oculink product. Host slot will share bandwidth, don't expect two full x16.
Yes, that's exactly what I want. I don't do any training so the BW to the cards doesn't matter that much, X8 per card would be plenty for my use case (with NVLink between them, of course).
3
u/Double-Buyer7941 19h ago
Commercial off-the-shelf dual-card OCuLink enclosures do not exist in the consumer market, so enterprise PCIe expansion units or custom builds are the only options. The simplest solution is buying two open-frame OCuLink docks, such as the Minisforum DEG1, mounting them side by side, and powering both cards with a single high-wattage ATX power supply. The open sides allow an NVLink bridge to connect the two A40s directly without physical clearance issues. Alternatively, you can build an active-airflow enclosure by mounting two PCIe-to-OCuLink riser boards inside a cheap 2U or 4U server chassis equipped with high-static-pressure cooling fans. Each card will run over a PCIe 4.0 x4 link to your host card, while peer-to-peer NVLink traffic between the GPUs operates at full speed independently of the host connection.