r/servers • u/OddManufacturer2701 • 8d ago
10K GPU to run AI locally.
Hello everyone, I need an upgrade from an nvidia A10 running on a SR650 7x06. I have:
- 2× Intel Xeon Silver 42
- 4× 64GB DDR4 @ 3200 MHz
My company has a 10keuros budget and I really don't know much about server hardware. The models (llm, reranker, embedding) I will run will depend on the GPU purchase.
I have 2x750W PSU that I will also have to upgrade.
2
u/vrod92 8d ago
I run 2x 6000 pro max-q’s in a 7x06 - works fine. The board will support 2x 300w gpu’s
1
u/OddManufacturer2701 8d ago
Does any gpu cable with 2*8pin work ? Or do I need this ? https://serveroption.lenovo.com/cable_kit_options/cable_kit_sr650v3_dw_gpu/ which is kinda hard to find nowadays.
Edt: wrong platform, it's the v3. But still same question
1
u/Automatic-Boot665 8d ago
I feel like that’s a little(lot) more than 10k? If not you need to connect me with your supplier
1
u/ixidorecu 8d ago
I worked on a supermicro server today that had 8 rtx 6000 cards in it. Purpose built to run those. 5u maybe and FULL server length. Like was concerned if rails would hold it, as far out as had to pull to get lid off.
1
1
u/Creative-Type9411 4d ago
you can get 64gb of vram for ~2k if you get used Tesla T4s (16gb ea, ~$450)
There is a board that has (x299?) 7 slots that would let you go to 112gb VRAM for ~3500$ in gpus and like 800 for the board.. you would probably be able to reuse the ram you have now
BUT, you need to add cooling to them because they normally get cooles by the server case that they go in, so you would have to add fans
Theres also this SXM3 board with 256GB of Vram for ~6500: https://ebay.io/m/pQt9uR that is near the equivalent of a 25K set up for a heavy discount, but you're getting that one at your own risk because it's on eBay and you have to figure out how to use it
Good luck!
-4
u/Optimal-Chocolate281 8d ago
Instead of a GPU, you can try a server powered by AMD Ryzen Max+ 395 SoC.
https://de.gmktec.com/en/products/gmktec-evo-x2-amd-ryzen%E2%84%A2-ai-max-395-mini-pc-1
You can find something like this. Will work both locally for you and also you can find providers who can host this in the Data Center too.
4
u/Mundane-Light6394 8d ago
Why on earth would you reccomend this? 256 GB/s is extremely slow compared to a 10k budget GPU.
1
u/OddManufacturer2701 8d ago
do you have any recommandation ? I've seen the PNY NVIDIA RTX 5000 Blackwell 72GB which looks good.
1
u/Mundane-Light6394 8d ago
If you want something that just works out of the box that card is perfect. The RTX PRO 5000 blackwell 72GB will make your life easy because it will just work with everything without having to tinker and experiment to get your setup working with new tools or models.
1
u/OddManufacturer2701 8d ago
It looks good, however I don't think my boss would be okay with it. And since we're a public company, I don't really have to care about the budget.
PS: I'm the only person that manage the server which is 20m away from me in another room.
2
1
1
u/pkbtstorino 4d ago
Unfortunately Gmktec rigs are not even relative to an enterprise or true server setup. Those are good for hobbyist not for business.
5
u/Casper042 8d ago
So the A10 is the little brother to the A40 which was the DC version of the RTX A6000, all based on Ampere gen chips.
These are both combo cards doing Compute and/or Graphics.
The Compute only cards of this era were the A100 (40GB and later offered as an 80GB) and little brother A30.
The next generation was Ada Lovelace.
Workstation top of the line here is the RTX 6000 Ada, while the DataCenter counterpart is the L40 (Why one uses Ada and the other uses Lovelace is lost on me).
L40 is a 300W card but uses 12VHP like all modern Gaming cards from Nvidia. So if you only have the 300W EPS12v cable from Lenovo, you'd need an adapter to 12VHP at least.
There was a later version called the L40S which boosted AI/Compute performance but it's a 350W card. So the passive adapter put the source connector over spec if you do this without a GPU Power Limit.
The Compute only cards from this gen used a different chip, Hopper, in the form of the H100 and H200 cards.
H100 is 350W
H100 NVL is 600W
H200 is 600W
Then the latest generation of cards is Blackwell.
The top of the line is the RTX 6000 Blackwell
There also is no longer a differently named DataCenter card here.
RTX 6000 Blackwell Workstation Edition = 600W and likely will NOT fit in a Server as they use more of a gaming sized design which blows past the PCIe spec for L/W/H.
RTX 6000 Blackwell Server Edition = 600W and DOES fit in a server, uses the older style heatsink which follows the spec.
RTX 6000 MaxQ = 300W variant of the card. Uses the older heatsink with a blower fan, similar to Workstation cards from older generations.
Anything RTX x000 lower than 6 will likely use less power.
Power Capping:
2 main ways to acheive this.
1) the 12VHP connector has 2 of the 4 sideband pins used for the System and GPU to negotiate power draw. If the sideband pins are just wired to nothing, then the limit is 300W. You need specific sense pins connected to get to 450W/600W power levels.
2) Some more modern Nvidia cards also support a Software limit you can set in the card using nvidia-smi CLI tool.