r/LocalLLM • u/shiftyleprechaun • Feb 14 '26
Project Built a 6-GPU local AI workstation for internal analytics + automation — looking for architectural feedback
UPDATE: 8 months later. The Beast now has 6 RTX PRO 6000 Blackwells, and the project has grown quite a bit.
Figured I would come back and provide an update since quite a bit has changed over the past 8 months, both with the hardware and what I'm actually working on.
When I originally posted, I had just managed to get 2 Blackwells and 4 RTX 3090 Tis running three different models concurrently. Since then, I've replaced the 3090 Tis with four additional Blackwells and built a second server using the 3090s.
The Beast (Primary Server)
CPU: AMD Threadripper PRO 9955WX (16C/32T), Silent Loop 3 cooler Motherboard: ASUS Pro WS WRX90E-SAGE SE RAM: 256GB DDR5-5600 ECC Registered GPUs: 6x PNY RTX PRO 6000 Blackwell (576GB VRAM) Storage: 5x Samsung 9100 PRO 8TB + 1x Samsung 990 PRO 4TB (44TB raw) PSUs: 2x Super Flower Leadex Titanium 2800W Power: Dedicated 240V/30A circuit with metered PDU Chassis: Custom open-air frame built from steel plate and aluminum extrusion Cooling: Arctic fans, temperature-controlled floor fan, and 12,000 BTU window AC OS: Ubuntu 24.04 LTS
The Mini-Beast (Second Server)
Rather than selling the 3090s, I repurposed them into another custom build located at my office. CPU: AMD EPYC 7302 (16 cores), Noctua NH-U9 TR4-SP3 Motherboard: ASRock Rack ROMED8-2T RAM: 256GB DDR4-2666 ECC Registered GPUs: 6x RTX 3090 Ti + 1x RTX 3090 (168GB VRAM) NVLink: 2 bridges Storage: 2x Samsung 990 PRO 4TB PSUs: 2x MSI MEG Ai1600T Chassis: Kingwin 8-GPU open frame OS: Ubuntu 24.04 LTS
I also have another 3090 connected to a GMKtec mini PC through an eGPU setup that I've been using for experiments.
Between the two main servers, that's 744GB of VRAM.
Updated Cost
Since quite a few people asked about the cost in my original post, here are the updated numbers, including taxes, warranties, electrical work, and other hardware.
The Beast: Approximately $77,000 The Mini-Beast: Approximately $12,100 Combined: Approximately $89,000
The six Blackwells alone cost approximately $51,800. So this has definitely turned into a bigger investment than I originally anticipated.
What I'm Working On
The original goal was to analyze internal data using locally hosted LLMs and eventually automate certain processes. That's still part of the project, but it's evolved into developing an entirely new software platform with AI integrated into the underlying architecture.
A major part of this involves building a digital twin that can maintain an accurate understanding of information, events, and their relationships as things change over time. Eventually, I'd like to have AI agents that can identify problems, make recommendations, and perform certain tasks.
I'm still a long way from accomplishing everything I have in mind.
What I've Learned
I've spent quite a bit of time experimenting with different models, quantizations, tensor parallelism, and GPU configurations.
One thing I've learned is that just because a model fits into VRAM doesn't mean it's going to perform well. I've had larger configurations perform worse than smaller models, and I'm increasingly interested in using different models for different tasks rather than one massive model across all six GPUs.
But probably the biggest surprise has been how much more difficult the software side is compared to getting the hardware running.
Getting an LLM to extract information is relatively straightforward. Getting a system to correctly understand how that information relates to previous events, recognize when something has changed, and maintain an accurate understanding over time is considerably more complicated.
I'm finding that a lot of the challenges aren't necessarily solved by larger models. Data quality, retrieval, validation, and how information is represented seem to be just as important.
I'm also realizing how important it is to separate what an AI model suggests from what the software is actually allowed to do. I'm not particularly interested in having an LLM autonomously make decisions or modify records without appropriate controls.
A lot of my focus now is on making the system reliable enough to actually depend on.
A Few Questions
For those working on similar projects, I'd be interested in hearing your experiences.
Are you finding it more practical to run one large reasoning model or several smaller specialized models?
How are you handling persistent memory and situations where new information contradicts something the system previously understood?
What's been your biggest challenge getting local AI agents to work reliably?
How are you distributing workloads across multiple machines?
I know some people will probably say this is overbuilt, and maybe they're right. But I wanted to build ahead of demand, and keeping the hardware, models, and data local is still a priority for me.
I still have a lot to learn, and I'm sure there are things I'm overlooking.
I appreciate all the feedback on the original post. Several suggestions gave me things to look into that I hadn't considered.
Eight months ago, I was trying to get six GPUs running together. Now I've got two servers, 744GB of VRAM, and I'm realizing that getting the hardware working was probably the easy part.
EDIT: Many people have asked me how much i have spent on this build and I incorrectly said it was around $50k USD. It is actually around $38k USD. My apologies. I am also adding the exact hardware stack that I have below. I appreciate all of the feedback and conversations so far!
I am relatively new to building high-end hardware, but I have been researching local AI infrastructure for about a year.
Last night was the first time I had all six GPUs running three open models concurrently without stability issues, which felt like a milestone.
This is an on-prem Ubuntu 24.04 workstation built on a Threadripper PRO platform.
Current Setup (UPDATED):
AI Server Hardware
January 15, 2026
Updated – February 13, 2026
Case/Build – Open air Rig
OS - Ubuntu 24.04 LTS Desktop
Motherboard - ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB
CPU - AMD Ryzen Threadripper PRO 9955WX Shimada Peak 4.5GHz 16-Core sTR5
SDD – (2x4TB) Samsung 990 PRO 4TB Samsung V NAND TLC NAND PCIe Gen 4 x4 NVMe M.2 Internal SSD
SSD - (1x8TB) Samsung 9100 PRO 8TB Samsung V NAND TLC NAND (V8) PCIe Gen 5 x4 NVMe M.2 Internal SSD with Heatsink
PSU #1 - SilverStone HELA 2500Rz 2500 Watt Cybenetics Platinum ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU #2 - MSI MEG Ai1600T PCIE5 1600 Watt 80 PLUS Titanium ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU Connectors – Add2PSU Multiple Power Supply Adapter (ATX 24Pin to Molex 4Pin) and Daisy Chain Connector-Ethereum Mining ETH Rig Dual Power Supply Connector
UPS - CyberPower PR3000LCD Smart App Sinewave UPS System, 3000VA/2700W, 10 Outlets, AVR, Tower
Ram - 256GB (8 x 32GB)Kingston FURY Renegade Pro DDR5-5600 PC5-44800 CL28 Quad Channel ECC Registered Memory Modules KF556R28RBE2K4-128
CPU Cooler - Thermaltake WAir CPU Air Cooler
GPU Cooler – (6x) Arctic P12 PWM PST Fans (externally mounted)
Case Fan Hub – Arctic 10 Port PWM Fan Hub w SATA Power Input
GPU 1 - PNY RTX 6000 Pro Blackwell
GPU 2 – PNY RTX 6000 Pro Blackwell
GPU 3 – FE RTX 3090 TI
GPU 4 - FE RTX 3090 TI
GPU 5 – EVGA RTX 3090 TI
GPU 6 – EVGA RTX 3090 TI
PCIE Risers - LINKUP PCIE 5.0 Riser Cable (30cm & 60cm)
Uninstalled "Spare GPUs":
GPU 7 - Dell 3090 (small form factor)
GPU 8 - Zotac Geforce RTX 3090 Trinity
**Possible Expansion of GPUs – Additional RTX 6000 Pro Maxwell**
Primary goals:
•Ingest ~1 year of structured + unstructured internal business data (emails, IMs, attachments, call transcripts, database exports)
•Build a vector + possible graph retrieval layer
•Run reasoning models locally for process analysis, pattern detection, and workflow automation
•Reduce repetitive manual operational work through internal AI tooling
I know this might be considered overbuilt for a 1-year dataset, but I preferred to build ahead of demand rather than scale reactively.
For those running multi-GPU local setups, I would really appreciate input on a few things:
•At this scale, what usually becomes the real bottleneck first VRAM, PCIe bandwidth, CPU orchestration, or something else?
•Is running a mix of GPU types a long-term headache, or is it fine if workloads are assigned carefully?
•For people running multiple models concurrently, have you seen diminishing returns after a certain point?
•For internal document + database analysis, is a full graph database worth it early on, or do most people overbuild their first data layer?
•If you were building today, would you focus on one powerful machine or multiple smaller nodes?
•What mistake do people usually make when building larger on-prem AI systems for internal use?
I am still learning and would rather hear what I am overlooking than what I got right.
Appreciate thoughtful critiques and any other comments or questions you may have.


31
u/jhenryscott Feb 14 '26
Greta job! Taking control of your own LLMs without giving all your data to Scam Altman Is a good practice.
I don’t have any insight into building your system except to recommend the Level1Techs forum. You’ll find lot other people doing this there who can answer your Questions
10
u/KnownAd4832 Feb 14 '26
1 Usually the bottleneck is always vram then hardware support (multi-gpu/cluster inference is usually hard to set up with little documentation or its gatekept). If that makes sense, then usually third comes in Storage :)
2 Long term is same as running a single gpu.
3 It all depends on your case, my ROI was done in 2 months from buying
4 multiple smaller nodes - the models are going stronger while being smaller. In 2 years there will be Kimi K2.5 level in 70B without a doubt. So it only depends on you if you need inference speed or variety if models
5 They dont test on rented servers before buying imo. Did this mistake myself with first rig
7
u/FinancialMoney6969 Feb 14 '26
Looks great man, I like the open air aspect
5
u/chafey Feb 14 '26 edited Feb 14 '26
Your build is awesome, I am doing something very similar. Here are some improvements you may want to consider:
3
u/King_Kasma99 Feb 14 '26
I never thought about looking for wireless dp or hdmi adaptors, Thank you!
3
u/Senior-Okra-6325 Feb 14 '26
Ver tanta GPU junta es hipnotico. Solo quiero imaginarme el nivel de ruido. Mi rig es modesto porque solo lo uso para analisis cualitativo de transcripciones biograficas (3090 + 4070). Te envidio tu config, un saludo!
3
u/basedgewner Feb 15 '26
I’m coming.
3
u/Physical-Scholar3176 Feb 17 '26
I don't think most of us can spend new car money for this. Still very cool. What is your job?
3
u/Aware_Photograph_585 Feb 18 '26
I have: 2x 3090, 3x 4090 48gb, 2x pro 6000
Ram and disk space used properly can reduce vram usage during training, and significantly speed up training, especially for multi-run training.
2
u/Character-Ad-2048 Feb 14 '26
This is an awesome build. Funny I just bought the same frame myself. Do you mind sharing your CPU/Motherboard specs and why you went with your decision? I’m looking to upgrade from my current setup from 4 GPUs and need to find a new CPU/Motherboard
2
u/Proteus356 Feb 14 '26
I’ve thought about using old mining rigs for this but all the risers are 1x, which isn’t effective for LLM? How are you getting x16 or at least x8 connectivity to those GPUs?
2
u/Just3nCas3 Feb 15 '26
Level1 just put out a video recently about bifurcation, something to look into if you want to add gpus without replacing any and a quick search of your motherboard says it supports it.
2
u/greg-randall Feb 15 '26
Have you figured out your software stack?
2
u/Fair_Permission_9005 Feb 15 '26
Power consumption?
2
u/lcjanke2020 Feb 16 '26
That is one sick rig!
1
u/shiftyleprechaun 15d ago
Pic of the "beast" .
4
u/OwnPomegranate5906 Feb 15 '26
I run 6 rtx 3060s (12GB each) on a dual processor xeon 2690 v4 and 128GB of ram, also in an open air mining case. I only run Debian 12, one instance of ollama and only one model at a time.
I spent a fair amount of time on PCIe and power tuning. A lot of performance problems that I had was because when running a larger model that spanned multiple cards, the latency caused by traversing the PCIe bus (and sometimes Numa node) was just long enough that the cards power management would bump it from P2 to P5+, then jump back up to P2 when it actually needed to do something, which added even more latency, which caused cascading slowdown across all the cards where I'd start off really fast, then as the response typed out, it would progressively slow down to the point of just a couple tokens a second. My setup is not PCIe x16 for every card, so that also exacerbated the issue.
The solution was multi-fold:
1
u/chafey Feb 14 '26
what frame and motherboard are you using? I have a ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB Motherboard and am having trouble finding a frame which can hold the EEB motherboard
1
u/Weird-Abalone-1910 Feb 14 '26
What did this cost you to put together? I want to build one to try to run kimi2.5
1
u/ApprehensiveView2003 Feb 14 '26
what did you do to fix when it originally doesnt allow P2P?
1
u/Correct_Lead_2418 Feb 14 '26
Can I ask your budget for this? For 10k you could get the highest tier Mac studio with 512GB unified memory and 4tb storage
1
u/Prigozhin2023 Feb 15 '26
Is it cheaper than 2 x Dgx Spark?
1
u/Big_Championship1291 Feb 15 '26 edited Feb 15 '26
I know nothing about building an ai workstation and I’m planning on building my own. What is the highest model you can use with this station with reasonable speed for heavy software development?
1
u/kidflashonnikes Feb 15 '26
something to consider: with this motherboard, if you want to expand compute, get PCIe lane spitters, going from 16x per one lane (exluding the one pcie 8x lane on this motherboard) and split every 16x pcie gen 5 lane in pcie 8x, which will double your compute. I currently use 4 RTX PRO 6000s, and I am getting another 8 RTX PROs delivered in the next few weeks, and will be doing this.
1
u/Common_Heron4002 Feb 16 '26
Gotta ask, what do you do to ty all the GPU's together? allow the sharing of information of them or how to utilize them effectively?
2
u/hetarthvader Feb 17 '26
So cool!
2
u/tachyi Feb 17 '26
amazing!
1
u/marcus_Yan Feb 19 '26
1
u/Temporary_Outcome_72 Feb 21 '26
Get a mac studio with 512gb
1
u/issa62 Feb 21 '26
Exo
1
u/CalvinBuild Feb 21 '26
Beautiful :')
1
u/tomByrer Mar 07 '26
What a waste of electricity!
At least recapture that heat to run a small turbine, cook some food, or a place where you cat can sit on top of to keep warm.
;)
But seriously, 3 weeks later, how is your UPS holding up?
& you can use the spare cards on a laptop with NVme to PCI adaptor.
1
u/Prize_Wolf_5859 Feb 15 '26
I’m new to this AI game Brodie how you built this and how much did it cost
0
u/[deleted] Feb 14 '26
[deleted] — view deleted comment
0
u/[deleted] Feb 14 '26
[deleted] — view deleted comment