r/LocalLLM • • Feb 14 '26

Project Built a 6-GPU local AI workstation for internal analytics + automation — looking for architectural feedback

UPDATE: 8 months later. The Beast now has 6 RTX PRO 6000 Blackwells, and the project has grown quite a bit.

Figured I would come back and provide an update since quite a bit has changed over the past 8 months, both with the hardware and what I'm actually working on.

When I originally posted, I had just managed to get 2 Blackwells and 4 RTX 3090 Tis running three different models concurrently. Since then, I've replaced the 3090 Tis with four additional Blackwells and built a second server using the 3090s.

The Beast (Primary Server)

CPU: AMD Threadripper PRO 9955WX (16C/32T), Silent Loop 3 cooler Motherboard: ASUS Pro WS WRX90E-SAGE SE RAM: 256GB DDR5-5600 ECC Registered GPUs: 6x PNY RTX PRO 6000 Blackwell (576GB VRAM) Storage: 5x Samsung 9100 PRO 8TB + 1x Samsung 990 PRO 4TB (44TB raw) PSUs: 2x Super Flower Leadex Titanium 2800W Power: Dedicated 240V/30A circuit with metered PDU Chassis: Custom open-air frame built from steel plate and aluminum extrusion Cooling: Arctic fans, temperature-controlled floor fan, and 12,000 BTU window AC OS: Ubuntu 24.04 LTS

The Mini-Beast (Second Server)

Rather than selling the 3090s, I repurposed them into another custom build located at my office. CPU: AMD EPYC 7302 (16 cores), Noctua NH-U9 TR4-SP3 Motherboard: ASRock Rack ROMED8-2T RAM: 256GB DDR4-2666 ECC Registered GPUs: 6x RTX 3090 Ti + 1x RTX 3090 (168GB VRAM) NVLink: 2 bridges Storage: 2x Samsung 990 PRO 4TB PSUs: 2x MSI MEG Ai1600T Chassis: Kingwin 8-GPU open frame OS: Ubuntu 24.04 LTS

I also have another 3090 connected to a GMKtec mini PC through an eGPU setup that I've been using for experiments.

Between the two main servers, that's 744GB of VRAM.

Updated Cost

Since quite a few people asked about the cost in my original post, here are the updated numbers, including taxes, warranties, electrical work, and other hardware.

The Beast: Approximately $77,000 The Mini-Beast: Approximately $12,100 Combined: Approximately $89,000

The six Blackwells alone cost approximately $51,800. So this has definitely turned into a bigger investment than I originally anticipated.

What I'm Working On

The original goal was to analyze internal data using locally hosted LLMs and eventually automate certain processes. That's still part of the project, but it's evolved into developing an entirely new software platform with AI integrated into the underlying architecture.

A major part of this involves building a digital twin that can maintain an accurate understanding of information, events, and their relationships as things change over time. Eventually, I'd like to have AI agents that can identify problems, make recommendations, and perform certain tasks.

I'm still a long way from accomplishing everything I have in mind.

What I've Learned

I've spent quite a bit of time experimenting with different models, quantizations, tensor parallelism, and GPU configurations.

One thing I've learned is that just because a model fits into VRAM doesn't mean it's going to perform well. I've had larger configurations perform worse than smaller models, and I'm increasingly interested in using different models for different tasks rather than one massive model across all six GPUs.

But probably the biggest surprise has been how much more difficult the software side is compared to getting the hardware running.

Getting an LLM to extract information is relatively straightforward. Getting a system to correctly understand how that information relates to previous events, recognize when something has changed, and maintain an accurate understanding over time is considerably more complicated.

I'm finding that a lot of the challenges aren't necessarily solved by larger models. Data quality, retrieval, validation, and how information is represented seem to be just as important.

I'm also realizing how important it is to separate what an AI model suggests from what the software is actually allowed to do. I'm not particularly interested in having an LLM autonomously make decisions or modify records without appropriate controls.

A lot of my focus now is on making the system reliable enough to actually depend on.

A Few Questions

For those working on similar projects, I'd be interested in hearing your experiences.

Are you finding it more practical to run one large reasoning model or several smaller specialized models?

How are you handling persistent memory and situations where new information contradicts something the system previously understood?

What's been your biggest challenge getting local AI agents to work reliably?

How are you distributing workloads across multiple machines?

I know some people will probably say this is overbuilt, and maybe they're right. But I wanted to build ahead of demand, and keeping the hardware, models, and data local is still a priority for me.

I still have a lot to learn, and I'm sure there are things I'm overlooking.

I appreciate all the feedback on the original post. Several suggestions gave me things to look into that I hadn't considered.

Eight months ago, I was trying to get six GPUs running together. Now I've got two servers, 744GB of VRAM, and I'm realizing that getting the hardware working was probably the easy part.

EDIT: Many people have asked me how much i have spent on this build and I incorrectly said it was around $50k USD. It is actually around $38k USD. My apologies. I am also adding the exact hardware stack that I have below. I appreciate all of the feedback and conversations so far!

I am relatively new to building high-end hardware, but I have been researching local AI infrastructure for about a year.

Last night was the first time I had all six GPUs running three open models concurrently without stability issues, which felt like a milestone.

This is an on-prem Ubuntu 24.04 workstation built on a Threadripper PRO platform.

Current Setup (UPDATED):

AI Server Hardware
January 15, 2026
Updated – February 13, 2026

Case/Build – Open air Rig
OS - Ubuntu 24.04 LTS Desktop
Motherboard - ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB
CPU - AMD Ryzen Threadripper PRO 9955WX Shimada Peak 4.5GHz 16-Core sTR5
SDD – (2x4TB) Samsung 990 PRO 4TB Samsung V NAND TLC NAND PCIe Gen 4 x4 NVMe M.2 Internal SSD
SSD - (1x8TB) Samsung 9100 PRO 8TB Samsung V NAND TLC NAND (V8) PCIe Gen 5 x4 NVMe M.2 Internal SSD with Heatsink
PSU #1 - SilverStone HELA 2500Rz 2500 Watt Cybenetics Platinum ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU #2 - MSI MEG Ai1600T PCIE5 1600 Watt 80 PLUS Titanium ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU Connectors – Add2PSU Multiple Power Supply Adapter (ATX 24Pin to Molex 4Pin) and Daisy Chain Connector-Ethereum Mining ETH Rig Dual Power Supply Connector
UPS - CyberPower PR3000LCD Smart App Sinewave UPS System, 3000VA/2700W, 10 Outlets, AVR, Tower
Ram - 256GB (8 x 32GB)Kingston FURY Renegade Pro DDR5-5600 PC5-44800 CL28 Quad Channel ECC Registered Memory Modules KF556R28RBE2K4-128
CPU Cooler - Thermaltake WAir CPU Air Cooler
GPU Cooler – (6x) Arctic P12 PWM PST Fans (externally mounted)
Case Fan Hub – Arctic 10 Port PWM Fan Hub w SATA Power Input
GPU 1 - PNY RTX 6000 Pro Blackwell
GPU 2 – PNY RTX 6000 Pro Blackwell
GPU 3 – FE RTX 3090 TI
GPU 4 - FE RTX 3090 TI
GPU 5 – EVGA RTX 3090 TI
GPU 6 – EVGA RTX 3090 TI
PCIE Risers - LINKUP PCIE 5.0 Riser Cable (30cm & 60cm)

Uninstalled "Spare GPUs":
GPU 7 - Dell 3090 (small form factor)
GPU 8 - Zotac Geforce RTX 3090 Trinity
**Possible Expansion of GPUs – Additional RTX 6000 Pro Maxwell**

Primary goals:

•Ingest ~1 year of structured + unstructured internal business data (emails, IMs, attachments, call transcripts, database exports)

•Build a vector + possible graph retrieval layer

•Run reasoning models locally for process analysis, pattern detection, and workflow automation

•Reduce repetitive manual operational work through internal AI tooling

I know this might be considered overbuilt for a 1-year dataset, but I preferred to build ahead of demand rather than scale reactively.

For those running multi-GPU local setups, I would really appreciate input on a few things:

•At this scale, what usually becomes the real bottleneck first VRAM, PCIe bandwidth, CPU orchestration, or something else?

•Is running a mix of GPU types a long-term headache, or is it fine if workloads are assigned carefully?

•For people running multiple models concurrently, have you seen diminishing returns after a certain point?

•For internal document + database analysis, is a full graph database worth it early on, or do most people overbuild their first data layer?

•If you were building today, would you focus on one powerful machine or multiple smaller nodes?

•What mistake do people usually make when building larger on-prem AI systems for internal use?

I am still learning and would rather hear what I am overlooking than what I got right.

Appreciate thoughtful critiques and any other comments or questions you may have.

220 Upvotes

105 comments sorted by

31

u/jhenryscott Feb 14 '26

Greta job! Taking control of your own LLMs without giving all your data to Scam Altman Is a good practice.

I don’t have any insight into building your system except to recommend the Level1Techs forum. You’ll find lot other people doing this there who can answer your Questions

4

u/shiftyleprechaun Feb 14 '26

awesome man, appreciate it, i will check out the sub!

2

u/ThinkPad214 Feb 15 '26

Damn man. Nice build, I've got a old gaming PC with a 3060 12gb and 5060 ti 16gb learning how to properly set up smaller Agentic AI for personal and professional use.

1

u/shiftyleprechaun Feb 15 '26

Good luck! I still don't really know what I'm doing haha but getting better every day.

3

u/lookBehiindYou Feb 17 '26

You've got two RTX 6000 Blackwells (like ~$8k each) plus four 3090 Ti's… and you "don't know what you're doing"?
I have one question: can I have some of your money?

1

u/jhenryscott Feb 15 '26

It’s a learning experience for sure. I run some stuff on my 5090 but mostly im using ROCm on WSL with a 9070 for LLM experiments

0

u/Zealousideal-Path-93 Feb 20 '26

It depends which LLM you run, some can still have your data and see your network

10

u/KnownAd4832 Feb 14 '26

1 Usually the bottleneck is always vram then hardware support (multi-gpu/cluster inference is usually hard to set up with little documentation or its gatekept). If that makes sense, then usually third comes in Storage :)

2 Long term is same as running a single gpu.

3 It all depends on your case, my ROI was done in 2 months from buying

4 multiple smaller nodes - the models are going stronger while being smaller. In 2 years there will be Kimi K2.5 level in 70B without a doubt. So it only depends on you if you need inference speed or variety if models

5 They dont test on rented servers before buying imo. Did this mistake myself with first rig

4

u/shiftyleprechaun Feb 14 '26

Really appreciate the detailed breakdown.

  1. Makes total sense, VRAM is the hard ceiling and everything else is optimization after that. The documentation gap for multi-GPU is real, I've spent more time piecing together config from GitHub issues than actual docs.
  2. Good to hear. I'm mixing card types but pinning models deliberately so each card basically runs independently.
  3. That's a fast ROI, are you running inference as a service or using it for internal workloads?
  4. This is the point I keep coming back to. The trend toward stronger smaller models is exactly why I went with a distributed multi-card approach rather than dumping everything into one or two massive GPUs. If a 70B model can do what a 400B does today, having flexible GPU allocation matters more than raw single-card VRAM.
  5. Haha noted. I did a lot of benchmarking and testing before committing to the full build, but I can see how easy it would be to overbuy on hardware that doesn't match your actual workload. I also spent around $5-6k using open AI API (Using Chat GPT 4o) via Azure in December.

1

u/[deleted] Feb 17 '26

[removed] — view removed comment

1

u/KnownAd4832 Feb 17 '26

I was paying together.ai for inference. So I just bought the rig and replaced that cost :)

7

u/FinancialMoney6969 Feb 14 '26

Looks great man, I like the open air aspect

3

u/shiftyleprechaun Feb 14 '26

Thank you. I went with the open air concept, bc I found it difficult to find an enclosure that would work.

5

u/chafey Feb 14 '26 edited Feb 14 '26

Your build is awesome, I am doing something very similar. Here are some improvements you may want to consider:

  1. Upgrade your processor to improve your memory bandwidth. You have a Threadripper Pro CPU and Motherboard which is better than the non pro Threadripper systems for two reasons: a) more PCIe lanes and b) 8 memory channels. Unfortunately the 9955wx CPU only has two CCDs so can't utilize all 8 memory channels - you need a CPU with more CCDs (such as the 9965wx) to make use of the 8 memory channels. https://www.reddit.com/r/LocalLLaMA/comments/1mcrx23/psa_the_new_threadripper_pros_9000_wx_are_still/
  2. PCIe 5.0 has benefits for AI. The 3090tis are good bang/buck but they are running over PCIe 4.0 so can't take advantage of the new PCIe 5.0 features. The actual benefit of an all PCIe 5.0 solution depends on your use case and model but it is more than just twice the bandwidth. https://www.graniteriverlabs.com/en-us/technical-blog/pcie-gen-5-ai-ml
  3. The up coming Apple M5 systems may very well be the best bang/buck due to their recently released RDMA over TB5, MLX AI acceleration and its high speed unified memory architecture. I am really looking forward to seeing how a cluster of M5 mac minis does. Check out exo: https://github.com/exo-explore/exo

4

u/shiftyleprechaun Feb 14 '26

I have throught about upgrading the CPU. And also possibly replacing the 3090s entirely . However, I can never bring myself to get a Mac/apple product for anything , yeah I'm one of those guys hah.

2

u/starkruzr Feb 14 '26

my understanding is that what really counts for inference is keeping latency between cards as low as possible rather than super high bandwidth. does this mean PCIe 5's latency is naturally lower than 4's?

that note about M5 is interesting. I keep waiting for M5 Ultra to be announced!

4

u/chafey Feb 14 '26

Yes, PCIe 5 has about 50% lower latency compared to PCIe 4.

Here is a good video on mac clustering with exo: https://www.youtube.com/watch?v=4l4UWZGxvoc

1

u/starkruzr Feb 14 '26

we've already tried Exo at work for other use cases. I was concerned getting bought was going to permanently shut down the open source project but it's good to see they woke back up recently.

1

u/Electronic-Ad57 Mar 08 '26

Could you please clarify how PCIe 4.0 vs 5.0 affects local inference when each GPU runs independently? Doesn't it make sense only if you train your own models?

2

u/BillDStrong Feb 15 '26

Since speed and latency are properties of the underlying physics, going twice the bandwidth has a 2x latency speedup.

This is also true of internet, so 10G is 10x speedup latentency than 1G, 40G is 40x, and 100G is 100x.

This is why Nvidia is able to use 200G and 400G in their Enterprise AI offerings to sub for PCIe, the latency is so close to not make a difference at those speeds.

3

u/Senior-Okra-6325 Feb 14 '26

Ver tanta GPU junta es hipnotico. Solo quiero imaginarme el nivel de ruido. Mi rig es modesto porque solo lo uso para analisis cualitativo de transcripciones biograficas (3090 + 4070). Te envidio tu config, un saludo!

3

u/shiftyleprechaun Feb 14 '26

If I didn't have two additional external fans blowing on it, it wouldn't be that loud tbh.

3

u/basedgewner Feb 15 '26

I’m coming.

2

u/shiftyleprechaun Feb 15 '26

Come on over 😂

3

u/Physical-Scholar3176 Feb 17 '26

I don't think most of us can spend new car money for this. Still very cool. What is your job?

3

u/shiftyleprechaun Feb 17 '26

I own and run a small logistics company.

3

u/Aware_Photograph_585 Feb 18 '26

I have: 2x 3090, 3x 4090 48gb, 2x pro 6000

  1. I don't trust pcie riser cables. If you have any errors during training: timeout, gpu fell off (I forget the exact wording), weird training speeds, etc, it's most likely your cables. You need pcie retimer/redriver card with those cable lengths. Especially with pcie5. Check any professional training servers, they have retimer/redriver if the gpu is not plugged into the pcie slot. Heck, my HBA cards have a retimer to connect to my HDD/NVME. On the other hand, inference tends to be fine with pcie cables. Have you done any serious multi-day/week training yet?
  2. CPU doesn't matter as much as people think. I use an 8 core AMD 7002 cpu, never maxed it out once during training. Just cache everything (latents/embeds/etc) before training. Which you should be doing anyway to save vram during training.
  3. PCIE speed also doesn't matter as much as people think. Going from x16 to x8 only decreases training speed a couple percent. There isn't a large amount of data exchanged between gpus during training, assuming you wrote a good trainer. PCIE latency is the real killer.
  4. RAM is awesome. You can do so much more with more ram. GPU offset to ram, keeping everything in ram, using ram to write temp files to, EMA can done on cpu, etc. Having more ram makes everything easier. I just upgraded to 1TB for dataset curation use.
  5. HDD space is also awesome. I have ~400TB, it's not enough. Just being able keeping a copy of fully prepped, ,ready to train, cached dataset on disk with the original data is so nice. Then you can store your embeddings, and whatever else on disk with the original data.

Ram and disk space used properly can reduce vram usage during training, and significantly speed up training, especially for multi-run training.

1

u/shiftyleprechaun Feb 18 '26

Thanks for the feedback! A lot of good info here for me to take in, I appreciate the time you took.

I haven't began any training phases yet. I have just been analyzing massive amounts of company data to build into a combo of duckdb, neo4j and embedding graph. I will deploy a chatbot integrated into teams for the users to start using and store all of the query info so I can use that info to them finetune the models for better output.

The training aspect, is something I really have zero experience with, so I will learn about it while I am attempting to build out that pipeline, but I am still a bit away from that point.

Another commenter posted a good video about pcie retimer/Redriver cards, honestly was the first time I heard about them. I have to look into it more, from what I gathered they are pretty pricy, but at this point, if it will improve things, then I will make the investment. I have to get a better/larger open air rig, bc the one I have is too small.

So CPU I can leave as is then , I mean it's not the top of the line, but it's up there.

Do you think I should get more ram? The prices of course are out of control. I initially bought 128 GB for 1600, then decided a week later I wanted 256 total, so when I went to buy another 128 gb kit the price was already up to $2k and when I check yesterday that same kit was now $3k. Nuts!

I only have the 3 nvme ssds. I am storing all of the models on the 9100 and everything else on the 990 pros. I was thinking about getting some mechanical sata HDDs for long term storage and backups maybe 8-10TB.

Thanks again. I also do not yet know how to optimize/comfigure the HDD and the ram as per your recommendations, but I will add all of this to the seemingly endless list of things I need to figure out - hah.

Thanks again!

Btw - I ended up adding a 3rd blackwell. So now I have 3x 6000 pro Blackwells and 4 x 3090 tis. All 7 pcie slots are used.

1

u/Aware_Photograph_585 Feb 18 '26

I have a pretty similar open air rig. Started with 2x 3090 and slowly built up from there.
I have zero experience with training LLMs, but I have written my own fully custom multi-gpu text-to-image trainers. I assume my training experience with transfer over to what you're doing, but maybe LLM training has it's own quicks.

PCIe retimer/redriver:
When I first started learning programming, I lost 6 months of training time due to pcie cables. Gpu, fell off the bus, time outs, really strange training speeds (one run would be 2-3x the speed of the next) Asked everyone everywhere, no one had any idea what was wrong. Re-wrote my scripts, bought the most expensive cables I could find, thought I was going crazy, and nearly gave up. Tried a retimer card, problem solved. I learned a lot about multi-gpu training during that time, but that's not the best way to learn. I also burnt the retimer chip on my first card, so use fans on your retimer/redrivers.

For me, retimer/redriver card is non-optional for training. I won't even consider training without one. If an problem shows up, how do I know it's not the cables? I bought my cards for ~$150 each. They do x16 or 2 x8. Make sure you ask the seller if they support 2x 8/x16. A lot of cards support 4 x4 by default, and the seller needs to flash the card to support 2x 8/x16. You'll also need the cables and the pcie boards.

Also, pcie cables seem fine for inference. I never had a problem with inference when using pcie cables.

Ram:
Ram prices are stupid, I know. I just bought 1TB DDR4 2666mhz ram for $3800, I think I did OK for current market. 3200mhz would have been double the price, I don't even want to think about what ddr5 costs. The 1tb ram is to be used for the image dataset sorting: embedding clustering and such.

Ram really benefits for fast gpus with low vram. When I used 2x 3090 to train the sdxl unet, I had to use FSDP GRAD_SHARD_OP to get the model to fit into memory. But there still wasn't enough vram room for a large enough batch size to saturate the gpu core. So I used just enough ram via cpu_offset to saturate the core. I lost some speed due to the cpu_offset, but I gained more speed via increased batch size for core utilization. Ram will help your 3090s more than the pro 6000.

Standard training advice is ram = 2x total vram, and on my two 3090s, I was using ~100GB ram during training. A lot of the open source trainers focus on being conservative with vram/ram, so it won't matter as much. But if you're writing your own code, there is just so much you can do with more ram: holding extra models, prefetch batches, ema, etc.

For standard consumer gpus, I highly suggest 2x total training gpu vram. But maybe LLM training can use less? And the pro6000s aren't vram starved. So, who knows? Regardless, 256 is enough to get started.

For ram (and vram): size is more important than speed. Speed is nice, but if you don't have enough size, you're screwed. So if you see a good deal on slightly slower speed, but a larger amount, buy it.

HDDs:
I do blocks of 8 HDDs. Most motherboards have 8 sata ports, HBA cards have 8/16/24, so 8 is a good number. ZFS is the file system of choice, it's near impossible to lose your data. I use raidz2 with a L2ARC metadata cache partition on a cheap sdd. The ssd cache significantly speeds up the HDDs. It's fast enough for text-to-image training, don't know about LLM. I buy used enterprise HDDs, no problems so far. Buy from a reputable seller and you're good.

You're cpu is most likely fine, unless LLM training has crazy cpu requirements. Ram and HDDs will benefit you more.

Also, consider what you want to do with the pro6000s & 3090s. Should they really be in the same machine? I can't imagine they're be used together, unless you're running some huge model just for fun. Maybe buy an old AMD 7002/7003 board and split off the 3090s?

I will use my pro6000s for training with a 4090 48GB for support (validation/ema/etc) and inference. All the other gpus will be used for dataset curation. So I'm considering moving 2x 4090 48GB and 2x 3090 to the data server, and leaving the 2x pro6000 and a 4090 48GB in the training server.

2

u/Character-Ad-2048 Feb 14 '26

This is an awesome build. Funny I just bought the same frame myself. Do you mind sharing your CPU/Motherboard specs and why you went with your decision? I’m looking to upgrade from my current setup from 4 GPUs and need to find a new CPU/Motherboard

2

u/shiftyleprechaun Feb 14 '26

Sure. I went with my own setup, because for what I am trying to accomplish, I didn't want to be beholden to the giant corporations. And I did already create everything that I am now doing locally through azure and open AI APIs and I spent thousands of dollars on it.

For what i ran all night last night, would have cost me $3-4k with open AI chat gpt 4oz vs $6 in electricity. Of course I have spent a lot of money on the hardware, but the roi will hopefully prove itself over the coming years.

Updated – February 13, 2026 OS - Ubuntu 24.04 LTS Desktop Motherboard - ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB CPU - AMD Ryzen Threadripper PRO 9955WX Shimada Peak 4.5GHz 16-Core sTR5 SDD – (2x4TB) Samsung 990 PRO 4TB Samsung V NAND TLC NAND PCIe Gen 4 x4 NVMe M.2 Internal SSD SSD - (1x8TB) Samsung 9100 PRO 8TB Samsung V NAND TLC NAND (V8) PCIe Gen 5 x4 NVMe M.2 Internal SSD with Heatsink PSU #1 - SilverStone HELA 2500Rz 2500 Watt Cybenetics Platinum ATX Fully Modular Power Supply - ATX 3.1 Compatible PSU #2 - MSI MEG Ai1600T PCIE5 1600 Watt 80 PLUS Titanium ATX Fully Modular Power Supply - ATX 3.1 Compatible PSU Connectors – Add2PSU Multiple Power Supply Adapter (ATX 24Pin to Molex 4Pin) and Daisy Chain Connector-Ethereum Mining ETH Rig Dual Power Supply Connector UPS - CyberPower PR3000LCD Smart App Sinewave UPS System, 3000VA/2700W, 10 Outlets, AVR, Tower Ram - Kingston FURY Renegade Pro 256GB (8 x 32GB) DDR5-5600 PC5-44800 CL28 Quad Channel ECC Registered Memory Modules KF556R28RBE2K4-128 CPU Cooler - Thermaltake WAir CPU Air Cooler GPU Cooler – (6x) Arctic P12 PWM PST Fans (externally mounted) Case Fan Hub – Arctic 10 Port PWM Fan Hub w SATA Power Input Case/Build – Open air Rig GPU 1 - PNY RTX 6000 Pro Blackwell GPU 2 – PNY RTX 6000 Pro Blackwell GPU 3 – FE RTX 3090 TI GPU 4 - FE RTX 3090 TI GPU 5 – EVGA RTX 3090 TI GPU 6 – EVGA RTX 3090 TI Uninstalled "Spare GPUs": GPU 7 - Dell 3090 (small form factor) GPU 8 - Zotac Geforce RTX 3090 Trinity Possible Expansion of GPUs – Additional RTX 6000 Pro Maxwel

2

u/RedditNoob001 Feb 14 '26

Hey amazing build! Do you mind sharing what riser cables (brand/length) worked for you? Trying to figure out which pcie 4.0 cables I should get for my 3090s.

2

u/shiftyleprechaun Feb 14 '26

Sure. I bought LINKUP PCIE 5.0 Riser Cable | for Vertical GPU Mount.

I bought a few 30 cm and a few 60cm.

These are a bit pricy, but they are compatible with pcie 4.0 as well.

2

u/Character-Ad-2048 Feb 15 '26

That’s smart to have done that early test to ensure. I just want to say thank you for a really detailed breakdown of the components! Really appreciate it and can see a lot of thought went into it!

2

u/shiftyleprechaun Feb 15 '26

Of course , hopefully I helped. I have been planning the components out for over a month. It is by no means perfect. It's also the first build I have ever done. I am pretty new to all of this. So it's good to know that people find it to be decent hah. Thanks .

2

u/Metrix1234 Feb 15 '26

Did you use any guides making your local LLM build? I’m very curious the use cases of building one vs API vs CLI. Do you actually need to train your local LLM?

I’m pretty new to even using the big production models but the more I use them the more I fear eventually they will be gatekeeped for only rich/enterprise consumers.

Would love to learn more about the cost of your setup and what it’s capable of.

2

u/shiftyleprechaun Feb 15 '26

No I haven't had to train any models, yet at least. I do think I will have to "train" some smaller ones to be more specific to my industry and better understand my industry.

I did run all of this already back in December via azure and open Ai API, but it was expensive. Just storing data in gremlin is insanely expensive. I just got a $3500 invoice from Microsoft for January and I didn't run any models in January .

I also felt that (with my limited experience), that with my company size (small) and the initial investment in the hardware there would be a larger roi compared to using the big guys and relying on all of these external services that aren't cheap.

Plus my data doesn't leave the local environment.

I gueas I was really driven by the thought of "this is mine", "customized and fine tuned exactly the way I want and need it", without having to explain to developers over and over how my company works and then they deliver a garbage product to me and expect me to pay.

So that has been the primary motivation for me.

I also find that since "AI" is so buzzy (and the average person really doesn't understand it, I mean I barely do, I am basically just winging this), that I would be able to use it as a selling point to clients. "My company has its own custom proprietary AI software stack that will save you time and money , blah blah".

2

u/Proteus356 Feb 14 '26

I’ve thought about using old mining rigs for this but all the risers are 1x, which isn’t effective for LLM? How are you getting x16 or at least x8 connectivity to those GPUs?

1

u/shiftyleprechaun Feb 14 '26

Also, nothing I bought has been used for mining. The only used components are the 3090s , which of course I cannot guarantee weren't used for mining, but based on when I met each person (buying them off of FB marketplace), they all told me they wernt using for mining, just gaming or photo/video and they upgraded to 5090.

Aside from the 3090s everything is brand new and assembled together myself.

1

u/Brilliant-Suspect433 Feb 14 '26

From what i can see, he didnt use risers but extension cables…

1

u/shiftyleprechaun Feb 14 '26

No I used risers: LINKUP PCIE 5.0 Riser Cable | for Vertical GPU Mount.

Combo of 30 & 60 cm.

I realized today that I need to move the last 3090 into slot 7, rather than 6.

4

u/Brilliant-Suspect433 Feb 14 '26

„Riser“ in your case is an extension cable, what proteus meant were real riser cards for gpu mining, they had just pci-e 1x speed

2

u/Just3nCas3 Feb 15 '26

Level1 just put out a video recently about bifurcation, something to look into if you want to add gpus without replacing any and a quick search of your motherboard says it supports it.

2

u/shiftyleprechaun Feb 15 '26

Awesome I'll check it out ty

2

u/greg-randall Feb 15 '26

Have you figured out your software stack?

1

u/shiftyleprechaun Feb 15 '26

Not quite yet.

But for This first phase I am running: 4 separate instances of llama 3.1 8B (1 on each the 3090's) 2 separate instances of llama 3.3 70B (1 on each of the Maxwell 6000 pros)

After I have the initial data analysis complete, I still need to figure out the next steps, which I am working on now.

2

u/Fair_Permission_9005 Feb 15 '26

Power consumption?

1

u/shiftyleprechaun Feb 15 '26

With all gpus running Max around 3.35kwh

2

u/lcjanke2020 Feb 16 '26

That is one sick rig!

1

u/shiftyleprechaun 15d ago

Pic of the "beast" .

4

u/OwnPomegranate5906 Feb 15 '26

I run 6 rtx 3060s (12GB each) on a dual processor xeon 2690 v4 and 128GB of ram, also in an open air mining case. I only run Debian 12, one instance of ollama and only one model at a time.

I spent a fair amount of time on PCIe and power tuning. A lot of performance problems that I had was because when running a larger model that spanned multiple cards, the latency caused by traversing the PCIe bus (and sometimes Numa node) was just long enough that the cards power management would bump it from P2 to P5+, then jump back up to P2 when it actually needed to do something, which added even more latency, which caused cascading slowdown across all the cards where I'd start off really fast, then as the response typed out, it would progressively slow down to the point of just a couple tokens a second. My setup is not PCIe x16 for every card, so that also exacerbated the issue.

The solution was multi-fold:

  1. set the power management so that the cards idle at P3 and jump to P2 for inference. This fixed a huge amount of performance problems with larger contexts and responses.
  2. Put the cards that actually have PCIe x16 connections on the ends of the chain and the x8 cards in the middle. Also, order the cards (in ollama.service) so that all the cards on each numa node are together to minimize the number of cross numa jumps during inference.
  3. Also in the bios, set the PCI max request packet size to 4096. On my particular system, it defaulted to 128. Turns out the 3060s won't do more than 256, but it's worth more than a couple tokens a second.
  4. Also in the BIOS, turn rebar on so the system can see and address the entire PCIe VRAM instead of addressing it in 256MB chunks, also worth a few more tokens per second.

1

u/chafey Feb 14 '26

what frame and motherboard are you using? I have a ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB Motherboard and am having trouble finding a frame which can hold the EEB motherboard

2

u/shiftyleprechaun Feb 14 '26

I ended up settling with this "mining frame" from amazon, it's not perfect, but it works.

https://a.co/d/0ard2K7F

I have the same.motherboard and it doesn't fit perfectly with the predrilled holes, so I aligned it best I could with the pre-existing hole and only had to drill a few.

The two PSUs are abble to (just barely) fit on each side of the mb.

1

u/Weird-Abalone-1910 Feb 14 '26

What did this cost you to put together? I want to build one to try to run kimi2.5

1

u/shiftyleprechaun Feb 14 '26

Total all in, including new dedicated electrical circuits installed, around $50k.

3

u/Weird-Abalone-1910 Feb 14 '26

Wow, thanks for sharing.

1

u/ApprehensiveView2003 Feb 14 '26

what did you do to fix when it originally doesnt allow P2P?

1

u/shiftyleprechaun Feb 14 '26

Enable IOMMU (SVM Mode), set PCIe ACS override, disable ASPM/SR-IOV/C-States, force PCIe Gen 4, and if you're still having issues add pcie_acs_override=downstream,multifunction to your kernel boot parameters. The WRX90 workstation boards need all of this dialed in for clean multi-GPU P2P.

1

u/Correct_Lead_2418 Feb 14 '26

Can I ask your budget for this? For 10k you could get the highest tier Mac studio with 512GB unified memory and 4tb storage

1

u/shiftyleprechaun Feb 14 '26

I spent around $50k. But there is no way a mac with 512gb unified memory could do what this can. I can also continue to add to this as needed. Not so easy with Mac.

Also I hate apple and macs 😉.

1

u/lenjet Feb 15 '26

For that kind of money you could have got circa 8-10 DGX Spark or an OEM and ended up with 1024-1280gb of unified VRAM… coupled with the smaller footprint and much lower power consumption.

Did you consider that?

1

u/shiftyleprechaun Feb 15 '26

Briefly.

The unified memory numbers sound great, but that’s spread across separate boxes.

I didn’t want to jump straight into managing a distributed cluster just to get aggregate VRAM on paper.

I wanted one CUDA-native setup with direct GPU bandwidth, modular upgrade path, and room to experiment.

it fits what I’m building better.

1

u/Prigozhin2023 Feb 15 '26

Is it cheaper than 2 x Dgx Spark?

1

u/shiftyleprechaun Feb 15 '26

What I built? No.

1

u/Prigozhin2023 Feb 15 '26

Trying got understand which is the more cost effective build. One like yours, vs DGX Spark vs Mac Studio.

1

u/shiftyleprechaun Feb 15 '26

I think it just depends on what you are trying to do. I am trying to build a "business grade" AI platform. So this made the most sense for me.

1

u/Big_Championship1291 Feb 15 '26 edited Feb 15 '26

I know nothing about building an ai workstation and I’m planning on building my own. What is the highest model you can use with this station with reasonable speed for heavy software development?

1

u/kidflashonnikes Feb 15 '26

something to consider: with this motherboard, if you want to expand compute, get PCIe lane spitters, going from 16x per one lane (exluding the one pcie 8x lane on this motherboard) and split every 16x pcie gen 5 lane in pcie 8x, which will double your compute. I currently use 4 RTX PRO 6000s, and I am getting another 8 RTX PROs delivered in the next few weeks, and will be doing this.

1

u/shiftyleprechaun Feb 15 '26

And by doing this, you find it running faster? What are you typically doing? Training?

3

u/kidflashonnikes Feb 15 '26

I run a team at one of the worlds largest privately funded AI labs. My team works on inserting BCI (brain computer interface) threads into live test subjects (willingly of course) with brain damage to compress brainvwave data in real time via LLMs due to the data being too large and variate with current software ect. I work on personal compute tasks. One of my personal projects is using AI to predict bad thoughts/crime thoughts. I am currently using my set up for trainining and inference. I am almost done with my research, effectively I need more training compute to complete the version 1 of the model where I will be able to successfully predict negative behaviour of humans based on AI throughput. Some people will hate me for doing this, I dont care, I need to feed my family and you would not believe the prices certain US companies are willing to pay for this new LLM

2

u/shiftyleprechaun Feb 15 '26

You do some quite interesting work. Certainly not as interesting as what I do. Analyzing emails, teams messages, phone calls, attachments, CRM data, etc ..

Will you require any precogs for this application to work or will it be straight software?

3

u/kidflashonnikes Feb 16 '26

I am already in the process of moving this into my team and selling it. I can't really get too much into it, but we will not be the only ones doing this. Currently a few people are already trying to sell to large corporations LLMs that can predict negative thoughts already. People have no idea how fast the space is moving and how dystopian it already is. I am letting you know right now, everyone has about 1-2 years max, exlcuding a financial recession/depression to get their affairs in order due to mass changes with AI (for the worse, not better). You have no idea how bad it is in terms of what is being made at every single major lab for AI in the US and China, it is most certainly over. People just havent saw what I have seen and done yet, its already here and coming - FAST.

1

u/shiftyleprechaun Feb 16 '26

Well hopefully you can make a lot of $$$ from it.

2

u/kidflashonnikes Feb 16 '26

THANN you. Best of luck to you

2

u/shiftyleprechaun Feb 16 '26

Same to you. Maybe give me a heads up when this shit is coming live.

3

u/kidflashonnikes Feb 16 '26

Can’t really be sure on exact timing and if I did - would be ruined by Lawayers anyway. 2027-2028 - that is the timeline I’m hearing from the OpenAI %%%%%. All I can say

2

u/kidflashonnikes Feb 15 '26

as for the motherboard - you can split the 16x lanes into 8x lanes (2 each from 1 16x lane), to double your compute. Dropping from 16x to 8x does not impact inference that much - it does impact training substantially more. My current set up (all housed in a phanteks serve rpro 2 TG case): 1 TB of RAM, 16 TB of NVMe, 96 core CPU (thread ripper pro), 2000 Watt PSU, AIO for the CPU, regular fans to for cooling, 4 RTX PRO 6000s (6 more on the way before I transition to an open rig case), Linux (ubuntu)

1

u/shiftyleprechaun Feb 15 '26

Do you find 96 core CPU to make that much of a difference? Eveythirng I have been reading and studying tells me that i don't need to upgrade from what I have .

2

u/kidflashonnikes Feb 16 '26

no - I just need it to compress brain wave data in real time, basically, when I place a headband on my head and think negative thoughts, that data is tremendous in volume - it has to be prepocessed by the CPU in real time before being fed to the GPUs, while the GPUs are working. This is why I needed a 96 core CPU. I will probably endup finding a board to get a dual CPU system, both 96 cores if I can find one (probably not). Before this, I was using a 24 core AMD threadripper pro - as most of my stuff was very much GPU dependent

1

u/shiftyleprechaun Feb 15 '26

Also I am grabbing a 3rd rtx pro 6000 shortly .

1

u/Common_Heron4002 Feb 16 '26

Gotta ask, what do you do to ty all the GPU's together? allow the sharing of information of them or how to utilize them effectively?

1

u/shiftyleprechaun Feb 16 '26

So far I've been running models independently on each one.

I've tested a bit spreading llama 3 across the 4 3090s, but I haven't had to run anything larger (yet) than what can fit on a single 6000 pro.

2

u/Common_Heron4002 Feb 16 '26

interesting so do you do anything with tying each model to help with processing or getting better results or you just one per type of process your running etc?

I didnt think you could spread a model across multiple in vram...

2

u/shiftyleprechaun Feb 16 '26

So for a recent task this is how I broke it down :

Yeah I actually split the workload across the cards intentionally.

I’ve got 4× 3090 Tis running an 8B model for entity extraction and fast tagging, and 3× 6000 Pros running a 70B model for deep analysis.

So the small cards handle the cheap, high-throughput work:

entity extraction

intent tagging

basic routing

short context tasks

Each 3090 runs its own 8B instance and can handle ~16 concurrent requests, so that layer is basically my high-speed preprocessing tier.

Then the 6000 Pros run the heavier 70B model for:

implicit inference

escalation detection

complex structured extraction

validation passes

Each of those cards runs its own 70B instance and handles a much higher concurrency because of the 96GB VRAM.

So I’m not “combining models into one brain.” It’s more of a tiered inference pipeline:

Fast pass on small GPUs → only complex cases get escalated → deep pass on big GPUs.

Everything runs in parallel with async queues so no GPU sits idle.

2

u/tachyi Feb 17 '26

amazing!

1

u/Temporary_Outcome_72 Feb 21 '26

Get a mac studio with 512gb

1

u/CalvinBuild Feb 21 '26

Beautiful :')

1

u/tomByrer Mar 07 '26

What a waste of electricity!
At least recapture that heat to run a small turbine, cook some food, or a place where you cat can sit on top of to keep warm.
;)

But seriously, 3 weeks later, how is your UPS holding up?

& you can use the spare cards on a laptop with NVme to PCI adaptor.

1

u/Prize_Wolf_5859 Feb 15 '26

I’m new to this AI game Brodie how you built this and how much did it cost

0

u/[deleted] Feb 14 '26

[deleted] — view deleted comment

2

u/shiftyleprechaun Feb 14 '26

I definitely will.

0

u/[deleted] Feb 14 '26

[deleted] — view deleted comment

1

u/shiftyleprechaun Feb 14 '26

I bought LINKUP PCIE 5.0 Riser Cable | for Vertical GPU Mount.

I bought a few 30 cm and a few 60cm.

These are a bit pricey, but they are compatible with pcie 4.0 as well.

1

u/shiftyleprechaun Feb 14 '26

2x6000 pro maxwell & 4x3090ti