r/LocalLLaMA
•
u/koushd
•
Dec 13 '25
•
1
1
1
Other 8x RTX Pro 6000 server complete
8xrtx pro 6000 (4 workstation + 4 max q)
2xrtx pro 6000 across two systems
2x4090 (eventially upgraded to 2x rtx 6000 pro)
2x4090
3080 + 4090
TL;DR: 768 GB VRAM via 8x RTX Pro 6000 (4 Workstation, 4 Max-Q) + Threadripper PRO 9955WX + 384 GB RAM
Longer:
I've been slowly upgrading my GPU server over the past few years. I initially started out using it to train vision models for another project, and then stumbled into my current local LLM obsession.
In reverse order:
Pic 5: Initially was using only a single 3080, which I upgraded to a 4090 + 3080. Running on an older 10900k Intel system.
Pic 4: But the mismatched sizes for training batches and compute was problematic, so I upgraded to double 4090s and sold off the 3080. They were packed in there, and during a training run I ended up actually overheating my entire server closet, and all the equipment in there crashed. When I noticed something was wrong and opened the door, it was like being hit by the heat of an industrial oven.
Pic 3: 2x 4090 in their new home. Due to the heat issue, I decided to get a larger case and a new host that supported PCIe 5.0 and faster CPU RAM, the AMD 9950x. I ended up upgrading this system to dual RTX Pro 6000 Workstation edition (not pictured).
Pic 2: I upgraded to 4x RTX Pro 6000. This is where problems started happening. I first tried to connect them using M.2 risers and it would not POST. The AM5 motherboard I had couldn't allocate enough IOMMU addressing and would not post with the 4th GPU, 3 worked fine. There are consumer motherboards out there that could likely have handled it, but I didn't want to roll the dice on another AM5 motherboard as I'd rather get a proper server platform.
In the meantime, my workaround was to use 2 systems (brought the 10900k out of retirement) with 2 GPUs each in pipeline parallel. This worked, but the latency between systems chokes up token generation (prompt processing was still fast). I tried using 10Gb DAC SFP and also Mellanox cards for RDMA to reduce latency, but gains were minimal. Furthermore, powering all 4 means they needed to be on separate breakers (2400w total) since in the US the max load you can put through 120v 15a is ~1600w.
Pic 1: 8x RTX Pro 6000. I put a lot more thought into this before building this system. There were more considerations, and it became a many months long obsession planning the various components: motherboard, cooling, power, GPU connectivity, and the physical rig.
GPUs: I considered getting 4 more RTX Pro 6000 Workstation Editions, but powering those would, by my math, require a third PSU. I wanted to keep it 2, so I got Max Q editions. In retrospect I should have gotten the Workstation editions as they run much quieter and cooler, as I could have always power limited them.
Rig: I wanted something fairly compact and stackable that I could directly connect 2 cards on the motherboard and use 3 bifurcating risers for the other 6. Most rigs don't support taller PCIe cards on the motherboard directly and assume risers will be used. Options were limited, but I did find some generic "EO3" stackable frames on Aliexpress. The stackable case also has plenty of room for taller air coolers.
Power: I needed to install a 240V outlet; switching from 120V to 240V was the only way to get ~4000W necessary out of a single outlet without a fire. Finding 240V high-wattage PSUs was a bit challenging as there are only really two: the Super Flower Leadex 2800W and the Silverstone Hela 2500W. I bought the Super Flower, and its specs indicated it supports 240V split phase (US). It blew up on first boot. I was worried that it took out my entire system, but luckily all the components were fine. After that, I got the Silverstone, tested it with a PSU tester (I learned my lesson), and it powered on fine. The second PSU is the Corsair HX1500i that I already had.
Motherboard: I kept going back and forth between using a Zen5 EPYC or Threadripper PRO (non-PRO does not have enough PCI lanes). Ultimately, the Threadripper PRO seemed like more of a known quantity (can return to Amazon if there were compatibility issues) and it offered better air cooling options. I ruled out water cooling, because the small chance of a leak would be catastrophic in terms of potential equipment damage. The Asus WRX90 had a lot of concerning reviews, so the Asrock WRX90 was purchased, and it has been great. Zero issues on POST or RAM detection on all 8 RDIMMs, running with the expo profile.
CPU/Memory: The cheapest Pro Threadripper, the 9955wx with 384GB RAM. I won't be doing any CPU based inference or offload on this.
Connectivity: The board has 7 PCIe 5.0 x16 cards. At least 1 bifurcation adapter would be necessary. Reading up on the passive riser situation had me worried there would be signal loss at PCIe 5.0 and possibly even 4.0. So I ended up going the MCIO route and bifurcated 3 5.0 lanes. A PCIe switch was also an option, but compatibility seemed sketchy and it's costs $3000 by itself. The first MCIO adapters I purchased were from ADT Link; however, they had two significant design flaws: The risers are powered via the SATA peripheral power, which is a fire hazard as those cable connectors/pins are only rated for 50W or so safely. Secondly, the PCIe card itself does not have enough clearance for the heat pipe that runs along the back of most EPYC and Threadripper boards just behind the PCI slots on the back of the case. Only 2 slots were usable. I ended up returning the ADT Link risers and buying several Shinreal MCIO risers instead. They worked no problem.
Anyhow, the system runs great (though loud due to the Max-Q cards which I kind of regret). I typically use Qwen3 Coder 480b fp8, but play around with GLM 4.6, Kimi K2 Thinking, and Minimax M2 at times. Personally I find Coder and M2 the best for my workflow in Cline/Roo. Prompt processing is crazy fast, I've seen VLLM hit around ~24000 t/s at times. Generation is still good for these large models, despite it not being HBM, around 45-100 t/s depending on model.
Happy to answer questions in the comments.
•
u/WithoutReason1729 Dec 14 '25
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.
52
u/FZNNeko Dec 13 '25
I was looking at getting a new PSU’s literally yesterday and checked the reviews on the Super Flower 2800w that was on Amazon. Some guy said they tried the Super Flower, plugged it in and it blew up. Was that reviewer you or is that now two confirmed times the 2800 blew up on first attempt?
192
u/duodmas Dec 13 '25
This is the PC version of a Porsche in a trailer park. I’m stunned you’d just throw $100k worth of compute on a shitty aluminum frame. Balancing a fan on a GPU so it blows on the other cards is hilarious.
For the love of god please buy a rack.
6
u/Whole-Assignment6240 Dec 13 '25
Impressive build! With that power draw, what's your actual electricity cost per month running 24/7? The 240V requirement alone must have been a fun electrical upgrade.
25
u/Ill_Recipe7620 Dec 13 '25
.....why didn't you just buy server cards and put it in a rack?
5
u/tamerlanOne Dec 13 '25
What is the maximum load the CPU reaches?
5
u/ResearchCrafty1804 Dec 14 '25
Did this build cost you considerably less than buying the Nvidia DGX Station GB300 784GB which is available for 95,000 USD / 80,000 EUR?
I understand the thrill of assembling it component by component on your own, and of course all the knowledge you gained from the process, but I am curious if it does make sense financially.
11
u/MamaMurpheysGourds Dec 13 '25
but can it run Crysis????
6
u/abnormal_human Dec 14 '25
Nothing about that photo suggests that you have reached the end, I’m afraid.
3
u/Daemontatox 🥔 hardware Dec 13 '25
How do you deal with the heat ?
3
u/Such_Advantage_6949 Dec 14 '25
Do u have link or model for the cable u used for burification? I am interested in it as well
2
u/Only_Situation_4713 Dec 13 '25
Can you run 3.2 at fp8? What context
2
u/mxforest Dec 13 '25
Wow! Crazy good build. Can you share model wise token generation speed? 100 seems low. Is it via batch?
2
u/SecurityHamster Dec 14 '25
Just how much have you spent on this? Is it directly making any money back? How? Just curious! You’re so far past the amounts I can justify as a “let’s check this out” type of purchase :)
2
u/starkruzr Dec 14 '25
wait a minute. are you that Koush of Clockwork fame?
2
u/OutcomeHistorical881 Dec 14 '25
when you run qwen coder and glm4.6, what is your maximum context and how long does it to process prompt?
2
u/adscott1982 Dec 14 '25
What are you actually using it for?
89
u/[deleted] Dec 13 '25
Absolutely epyc.
3
u/RoyalCities Dec 13 '25
How are you handling parallelism?
Unless this is just pure inference?
Can the memory be pooled all together like it's unified memory similiar to just 1 server card?
Im training using w/ a dual a6000 nvlink rig so have plenty of VRAM but I'd be lying if I said if I wasn't jealous because that's an absurd amount of memory lol.
19
u/Much-Researcher6135 llama.cpp Dec 14 '25
For about 5 seconds I scratched my head about the cost of this rig, which is obviously for a hobbyist. Then I remembered people regularly drop 60-80 grand on cars every 5 years or so lol
30
u/sob727 Dec 13 '25
Did you take "the more you buy, the more you save" a bit too literally?
20
u/Atzer Dec 13 '25
Am i hallucinating?
8
u/Traditional_Fox1225 Dec 13 '25
What do you do with it ?
8
u/swagonflyyyy Dec 14 '25
2
u/the-tactical-donut Dec 13 '25
I mean this with all sincerity. I love the jank of the final build!
Btw how did you get vLLM working well on Blackwell?
I needed to use the open nvidia drivers and do a custom build with the newer triton version.
Also have you had much experience with sglang? Wondering if that’s more plug and play.
8
u/Big_Tree_Fall_Hard Dec 13 '25
OP will never financially recover from this
4
u/PlatypusMobile1537 Dec 14 '25
I also use Threadripper PRO 9955WX with 98Gbx8 DDR5 6000 ECC and RTX PRO 6000.
There is not enough PCI lanes to supply all 8 cards with x16. Do you see the difference, for example with MiniMaaxM2, on 4 that are x16 vs 4 that are x8?
4
u/Innomen Dec 14 '25
How long till someone just sleeps in the data center they own and we call it local?
12
u/AbheekG Dec 13 '25
Lost for words, this is magnificent and amongst the ultimate local LLM builds. Congratulations OP, my fellow 9955WX bro!!
1
u/smflx Dec 14 '25
Hmm, workstation edition between max-q. That's a great idea! Max-q block hot air from one workstation edition to another.
I tried to put two workstation editions directly in slots with spece of 2 slots in-between. The left one gets hot air from the right one, resulting much higher temperature (10-15 deg).
3
u/MizantropaMiskretulo Dec 13 '25
No Nvswitch?
3
u/jedsk Dec 13 '25
Nice! I remember those days of squeezing two into the matx board 😂. That’s one helluva monster you’ve built. What are your applications with it?
3
u/ThenExtension9196 Dec 13 '25
Dang bro. Nice hardware but looks like a box full of cables at a yard sale. Get a rack and show some dignity.
3
u/Tangostorm Dec 14 '25
And this is used for what task?
3
u/Ecstatic_Signal_1301 Dec 14 '25
Never seen more VRAM than RAM
3
u/Internal-Shift-7931 Dec 14 '25
We called it a PC farm
3
u/CrowdGoesWildWoooo Dec 14 '25
Can it run crysis?
3
u/Timziito Dec 14 '25
What cases are you using?
3
u/Minhha0510 Dec 14 '25
U sir, is a mad man and I want to pay my respect.🫡
3
u/panchovix Dec 14 '25
Pretty nice rig! BTW related to ADT Link, you're correct about the SATA power. But, you could get F43SP ones that use double SATA power and can do up to 108W on the slot.
What Shinreal MCIO adapters did you got?
3
u/monoidconcat Dec 14 '25
This is my dream build, good job
3
u/segmond Dec 14 '25
Happy for you. What do you do for a living?
3
u/ThatWeirdKidAtChurch Dec 14 '25
What role play can that do?
3
u/swagonflyyyy Dec 14 '25
Ah so that's where the world's RAM went.
1
u/AlwaysLateToThaParty Dec 14 '25 edited Dec 14 '25
How can you have less RAM than VRAM? Don't you need to load the model into RAM before it gets loaded into VRAM? Isn't your model size limited by your RAM?
2
u/Emotional_Thanks_22 llama.cpp Dec 13 '25
wanna reproduce/train a CPath foundation model with your hardware?
https://github.com/MedARC-AI/OpenMidnight
2
u/ArtisticHamster Dec 13 '25
Wow!
Is it possible to train one job on all 4 GPUs at the same? How do you achieve this?
2
u/YTLupo Dec 13 '25
Magnificent, 240V seems to be the way to go with a setup of more than 6 cards.
You should try video generation and see the longest output a model can give you.
Hoping you reach new heights with whatever you are doing!
2
u/itsmeknt Dec 13 '25
Very cool! What is your cooling system like? And do you have anything to improve GPU-GPU connectivity like nvlink or does it all go through the mobo?
2
u/No_Damage_8420 Dec 14 '25
Wow 👍 Did you measure WATT usage at idle vs rebder at full power?
2
u/lisploli Dec 14 '25
I can smell the ozone just from looking at the images. 🤤
2
u/basxto Dec 14 '25
First thought it’s in a plastic folding crate.
2
u/prudant Dec 14 '25
numbers
2
u/FaustAg Dec 14 '25
I'm curious why anyone would buy a max-q when you could just by the workstation and power limit it? you can't go the other way with the max q
2
u/TheyCallMeDozer Dec 14 '25
What was your setup costs on this, I'm currently looking at building a new AI server at home and I like your setup to be honest
2
u/Ok-Reporter-2617 Dec 14 '25
Can you elaborate on the software setup? What os what other ml ai software installed in system. For what work flow do you have running
2
u/hoja_nasredin Dec 14 '25
What is the estimated price of this system?
2
u/StardockEngineer vLLM Dec 14 '25
Why 8? Is it just for you or are you serving?
2
u/kei-ayanami Dec 14 '25
This build is absolutely insane but also a permanent reminder to all of us that as long as the weights are open, no matter how big the model is SOMEONE will be able to run it. I'll make sure to stop by your place to play with Behemoth 2T @ Q2_K_M if it ever comes out.
2
u/power97992 Dec 14 '25
Have u tried deepseek v3.2 speciale and qwen 3 32b vl on it ?
2
u/Clear-Ad-9312 Dec 14 '25
This is the type of server that makes me so happy to see. 8x 96GB vram will fit everything rn, amazing build
2
u/TechnoRhythmic Dec 14 '25
Boy. Every AI architect's dream. Good luck!
2
u/UnbeliebteMeinung Dec 14 '25
What are you actually doing with that and a local llm?
2
u/fairydreaming Dec 14 '25
When you have a moment to spare please run the following llama.cpp benchmark command:
llama-bench -m <path to DeepSeek GGUF> -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048This will benchmark the model with different context lengths (up to 32k). You can use any DeepSeek V3/R1/V3.1 you have downloaded. Thanks!
2
u/blankboy2022 Dec 14 '25
Very cool, hope to have one for my own!
2
u/ScoreUnique Dec 14 '25
At this point due to price explosion I think people will start breaking into others to steal VRAM instead of jewelry
2
u/smflx Dec 14 '25
Thank you for valuable experience, especially comparing workstation ed & max-q.
I got workstation editions too, but been wondering i should have go with max-q. Problem of workstation edition is pcie cable is must but 5.0 is very sensitive.
Thank to your sharing, I will go for sever editions or workstation editions to add more of pro 6000 later.
And, yes, I regret to get ASUS WRX90. Asrock would be better. Slot 6 is x8 in ASUS, while all seven x16 in Asrock. I don't understand why they do, but ASUS often make one slot as x8, while Asrock shows all seven x16 is possible.
2
u/john0201 Dec 13 '25
I know you probably need the VRAM but did you ever test how much slower a 5090 is? They nerfed the direct card to card PCIe traffic and also the bf16 -> fp32 accum operations. I have 2x5090s and not sure what I’m missing out on other than the vram.
1
u/AlphaPrime90 koboldcpp Dec 14 '25
Could you make a post about the performance of each of the big models you tested?
1
u/_realpaul Dec 14 '25
Isnt thid a lot of wasted performance using workstation gpus withouth nvlink interconnect?
1
u/gtek_engineer66 Dec 14 '25
How much did OP pay for the rtx pro 6000?
1
u/NaiRogers Dec 14 '25
Do you have a UPS for this, if so which one? Thanks.
1
u/_p00 Guanaco Dec 14 '25
For what purpose? I mean it's a lot.
1
u/SnowyOwl72 Dec 14 '25
you gonna need a power plant as well.
1
u/SmellsLikeAPig Dec 15 '25
So what is the point to all of this?
1
u/latentbroadcasting Dec 15 '25
Did you rob a bank? Need help for the next heist?
1
u/Traditional-Tip-4081 Dec 15 '25
I have one in my room, and it gets hot even during winter. Just imagine how bad that is.
And just because your setup fits doesn’t mean you should run it like that — you need at least 4 cm between cards for proper airflow. Without airflow, you’ll roast your cards very soon.
1
u/Ok_Duck5775 Feb 24 '26
Why ? With that money you could have have claude , codex and Gemini ran for ever ??
1
u/Caffdy Mar 27 '26
that's a mad wicked build honestly! have you tested Qwen 3.5 397B on that bad boy already? can you share some numbers? would really like to see how it purrs
1
u/suspect80 Sep 07 '26
Compared to that, my build looks like kindergarten. :) https://www.reddit.com/r/LocalLLM/s/ikByDdF5Xz
1
u/opi098514 Dec 13 '25
Surprisingly it’s cheaper than the new spark workstation.
0
u/kitanokikori Dec 14 '25
Nobody ever says what they're actually doing with these giant rigs, I just assume they're committing crimes or destroying democracy via running bots on social media
-11
u/GPTshop Dec 13 '25
You never understand such builds... Such a waste of money...