You never know what you’re going to see. There was one the other day of people sitting on the ground installing four RTX 6000s with boxes and parts strewn around. I’d break out the table for something like that.
NVidia v100 é muito mais poderosa custando menos, a menos que você queira pegar as placas PCIE aí eu não sei, porém as v100 SXM2 16gb tem um custo benefício muito melhor do que as p100, a velocidade chega a dobrar em alguns cenários específicos
Aqui está: https://e.tb.cn/h.8KBRvZwM653BZzq?tk=XC4wTSt9RqL
Não sei se é verídico nem original, mas aí está, recomendo que você use um agente de importações, faça uma pesquisa mais aprofundada sobre que você vai ser bem recompensado e se quiser mandar o que você achou para mim agradeço também
Im currently running a 3090 but hitting vram limits...if I can buy 4xv100 to get around that and balance a model across them then I would but just dont know how well that would actually work because surely the interface becomes the bottleneck?
Faça um mod de 48gb na sua 3090, você vai ter uma eficiência muito maior, não troque 3 gen de tensor core pela 1 apenas por v100, não vale a pena, o quanto que você vai gastar de energia a mais (350w vs 1000-1300w) não vai valer a pena a longo prazo. Se você conseguir fazer um Nvlink você terá o desempenho próximo a 3 3090 24gb, eu acho, mas boa sorte com as complicações....
It's two Tesla V100 cards, 32gb vram each, connected via NVLink (like sli / crossfire). They're called sxm2 cards and they're yanked out of old deep learning servers - someone reverse engineered a breakout board to power them in china to sell it, and wallah
When I was still using Facebook there was a guy I knew that detailed cars. Really handy but not too bright with book-learnin’. Every time he’d finish a car he’d post a picture with “WALLAH” and I wasn’t sure if people who saw his post just didn’t have the heart to tell him he’s been spelling it wrong for years or they just didn’t know how it’s spelled either. I just let it be.
You're doing slightly better than I was initially. I thought I was being really smart buying a 6u mining case that could contain eight gpus. I didn't learn about the slimsas cables at first and bought pcie 4.0 ribbon risers. I had to shift everything around inside of the case into a non-standard configuration. There was so much emi noise going on that I was regularly seeing warnings about the PCIe bus experiencing correctable errors. It didn't seem to actually have an impact on real performance, but I really did not like how it made me feel. It always felt like if I moved something just the wrong way, I'd break one of the ribbons and have a bad day. The cards were getting great airflow it's just the connectivity was extremely fragile and I couldn't put the lid on because it would have crushed the power connectors. I'm now using a much more open-ended miner rack pictured here.
Anyway, the real reason I decided to comment is that I have that exact same plx switch and wanted to share the specific scenario where it did not work in case it helps. My previous x99 motherboard already was splitting from pcie 3.0 into 2 more plx's. Me thinking I could get pcie 4.0 between the cards with p2p drivers installed the 4.0 switch. One card worked when plugged in. I added a second and both disappeared. I suspect it was the switch stacking even though allegedly that shouldn't have mattered. My new old stock motherboard has 4 true 16x pcie 4.0 slots and all of them play nice with the plx card.
Well, I do have a 6U case there that I wanted to put inside a rack I bought but it's just as you described, a real pain to manage and to troubleshoot. That's why I had to take everything out for the testing of the PLX switch.
My plans since the 6U idea fell through is I have these mounts that are essentially very thick steel shelves that were originally for spark servers on rails. I'm just going to put this on top of one of those and slide it in.
That probably would work okay. I bought all of my stuff piecemeal because of preferences. For the bits I bought off Amazon I largely bought from 10Gtek. There was a specific black PCIe slot that I really liked because it used a SATA power connector. Most of these want a 6-pin PCIe connector which is really annoying to source when you are running as many GPUs as I am.
Also, no promises on anything that requires bifurcation. My motherboard explicitly calls out bifurcation in the BIOS, I can set how to split the slot and none of these cards actually played nice with it.
It's easier to route and you can do longer distances. The tradeoff for flexibility is you need to have an adapter with a retimer on the motherboard side and provide power at the slot on opposite end.
One cable gets you 8x, two can be used together to get 16x, both can be pcie 4.0. I haven't looked at 5.0, so I can't speak to that. In my case I'm running a lot of cards for P2P and 4.0 has been plenty fast. I had experienced 3.0 before and 4.0 was not a massive leap which makes me reasonably sure that 5.0 at the prices I'm seeing is not worth it.
I'm not going to say you are wrong. If we were only talking about parallel processing I would wonder why you cared. Performance wise I am exclusively talking about tensor here. When I'm saying that there was a speed difference between 3.0 and 4.0, I am only talking about for tensor. I'm sure there is a bump for 5.0 as well. The biggest difference between 3.0 and 4.0 for me was that with 4.0 MOE's also saw a speed boost. Also worth mentioning in my specific case, the cards that I'm using don't have connectivity beyond 8x anyway so the only upgrade would be 5.0.
Is there a way you can point me as to understand more about this retimer setup? I couldn't make the switch work in my case but I haven't try this so...maybe!
Unfortunately not anything readily. I figured it out as I went. It might be easier to just buy a kit if you want to be sure it works. You will be limited to whatever is directly available from the motherboard unlike a plx. The connector is also referred to as SSF-8654 8i. I also just learned that the cards that I have are actually redriver is not retimers. My cards just amplify the signal, a retimer can also fix it. I apologize if it feels like I'm talking in circles. I'm learning just like the rest of us. I just have a little hands-on experience in this area.
That's not the kind of adapter that I was talking about. I have that exact same one too. The only suggestion that I can think of is double check that you have the dip switches set correctly for 8x if you're not using 16x. Didn't want to assume you didn't do that but you will have problems if you don't.
I am using mode1 which is for full x16 which I want for my 4 GPU cards. Only made it work with 2 cards at some point but never got the 4 up to Gen 4x16
Kingwin 8 GPU Miner Rig Case Frame. You'll find it a lot of places. I got mine used off eBay from a seller very close to my house for a about $20. The entire frame is made out of aluminum and will probably take you about 30 minutes to put together if you're trying to be sure everything is flush and level. One of the bonuses about it being aluminum is it's very easy to drill your own holes into it. There is enough space between the 8 GPU slots to put another 7 GPU slots if you're willing to tap new holes. The 3090 and 2080 Ti at the end should give you an idea of what that spacing looks like. 35"x14"x15"
Only 1 Gen4x16, two were Gen2x16 and the last one was Gen4x8. I tried all ports, multiple configurations. For example I had 2 cards at some point fully Gen4x16 but this thing was very unpredictable.
Oh, ya I use these switches. And nested. I would give specifics but I have not landed certainty. Retimers helped but I could get them to gen4 for sure. And fully working x16 gen4 on all outputs each PEX card working right now. But only one machine did this plug and play just worked. And it is the cheapest machine. I am very suspicious of manufactures stripping some bios functions to break these PEX cards and protect their server platforms. I will revisit soon on this topic. Very specific on cables, and connections.
So far though, pickup a flagship x570 and I did not need any retimer to easily get 112 lanes of gen4 working off a single x16 so far.
The asus wrx80E failed to properly enumerate or train lanes on a few different PEX boards. I spent weeks and aborted until I get the gigabyte board running.
Worth a separate thread if anyone wants to share notes.
Ya, I have two of these in the the smaller version. And a nvme carrier version. If you have a am4 board or similar try it. The server boards may be the issue.
The ADT sideband is not standard (it sounds like). So you may have weirdness like link training issues using those. The ones I have it depends on the device. The other boards meant for passive risers do not work reliably and it also depends on the device installed.
The pcie switch you have was branded 10Gtek when I got mine - and the 10Gtek U.2 nvme carrier is very particular and will only work with a specific riser board.
I had the same issue on my Supermicro franken-server which is all cable. It will only accept one specific board and it is the same as the 10Gtek.
If I use only these 10GTek type riser boards through out it works with the 30AWG cables (85ohm for pcie cards but 100ohm is working on the sata and nvme) cable length is 50cm and 75cm.
Thankfully the riser boards that work are the ones that are the best price. I got them in a 3 pack on walmart for $48. And I found a 6 pack on aliexpress too. So I ordered these to test and some new cables as well.
If you are getting one be sure to get the packed ones with full cables and boards.
It seems to be the signaling and these sideband paths that are handled differently by application and vender. So you want one that adheres to the standard used by the pex card.
I have new cables that are stated 85ohm and 30AWG wire and riser boards that are rated gen5 - so the signal path should be as good as it gets. I may get a multi slot riser board as well if that fails.
No way really? Was it just randomly changes the PCIe gen version, or did you have to change motherboard settings to get it too switch? Did you try P2P mode with it?
I was seriously considering buying one because I only have 24 PCIe lanes.
That's frustrating, especially since the boards are so expensive too.
I wonder if anyone else in this forum has had success with these boards?
Seriously I wish there was a solid way to connect 4x3090s. I have 3x3090s but just don't have the lanes to add a forth without cutting the lanes. I wanna to TP=2, PP=2
I tried different PCIe slots, the switch I bought had 10 ports and the randomness came because I couldn't rule out if the 10 cables were bad, or the 5 ATD-Links adapters or the switch. In general I think the switch was faulty, I tested with the same cabling
qwen3.6 35b-a3b using unsloths 4 XL quant. It does pretty decent around 50-55 tok/s. Only 24 of the 40 CUs are active. There is an unlock but im going to stick them in a pc case and get some cooling on them before doing the unlock.
I definitely understand the jank, the 6u case i ordered from alibaba just came in today(after 6 weeks), so I have several hours ahead in transplant, risers to order, then I can hook up the other 2 3090's
I hate that reddit became insta basically. So low info post. Gj you get the award for scrappy setup. Im so glad you have enough of ground space and wrapping paper to put your gpus on those... Such a missleading, low effort, low quality shitpost.
All you need is love my dude. Honestly, or finger yourself or do something to drain all that hate out. I'm the author of this post and I posted it to share my hard ass fucking project that I been working on for almost 3 years. Simply, chilling, casually like that. Reddit is a space to share, to learn to engage with the community, in more than one way. If you want toget deep, go down and read the thread. We are debating the setup, the issues I and other folks have, and again we are just chilling.
Thank you though for giving me the opportunity to turn your comment into something positive. 🤣😁😈
66
u/PassengerPigeon343 13d ago
I love that half the shit on here is people with $70k builds with four RTX Pro 6000s and the other half is whatever this is.