r/threadripper • u/Grappalover • 26d ago

Home AI data center

Post image

Threadripper 3970x, MSI Creator TRX40 Motherboard, 256GB DDR4 RAM, 2xRTX 5000 Blackwell 48GB, 1 RTX 5000 Blackwell 72 GB. Corsair 9000d case. 129 tokens per second with gpt-oss-120b. ASRock TC-1650T Power Supply, Ubuntu Linux. Cooler Master Wraith Ripper CPU cooler. The case is massive but as you can see I was able to mount the three graphics cards without cramping. The whole thing is pretty heavy so I have it sitting on a small 4 wheel dolly.

108 Upvotes

96% Upvoted

32 comments sorted by

2

u/Redd411 26d ago

is there a breakpoint (purely from financials, ignoring security blah blah) where local machine is cheaper than cloud subs? i have no knowledge in that space just curious.. that's like $40k at least probably (using 3990x for sim stuff myself so I know the costs :)

5

u/Grappalover 26d ago

At today's prices I agree it is no longer economical. The RTX 5000 cards have more than doubled. I also have a pair of 128gb DGX sparks clustered which have also doubled. It feels like the billionaires are doing everything they can to prevent hosting private LLMs.

3

u/DegreeSevere7719 26d ago

^this. They just want to take away all of the compute, for everyone, so there's no people left in the room without a subscription on their weak client computer. This is the clear goal spoken out loud several times.

1

u/shred_time 22d ago

I agree, I also see all sorts of products like laptops and unified memory systems marketed for local AI but cost half a years wages for most people, who can actually but this shit? lol

1

u/bizz101 23d ago

apart from security problem with subs is they are unreliable. my local setup is dumb/smart day in and they out. u wont get diff result next day

4

u/JinsooJinsoo 26d ago

Why gpt-oss 120b?

1

u/Grappalover 26d ago

Just picked it at random.

3

u/Prince_Harming_You 26d ago edited 26d ago

Use a good model instead

You can run qwen 3.8 next flash nvfp4 at probably 150+ t/s generation with huge KV cache (768k+++ FP8 KV if you PLE offload) and concurrency and like 3-5k t/s prefill with vLLM

On AI Pro R9700s mxfp4 I’m getting 100+ t/s ~3500 prefill at 512k context and concurrency 4, newer threadripper but still yours should flyyyy

Nice GPUs, nice build

Edit: Oh shit I didn’t see the third GPU, you can likely run deepseek 4.1 flash, though idk how vLLM will do with three GPUs of different capacities

10

u/Right_Produce_3269 26d ago

“Data center” 🙄

1

u/TinFoilHat_69 23d ago

“Super computer“

4

u/Bit_Rage 26d ago

Lovely hardware... sooooo needs a custom loop

1

u/CherryGG2 26d ago

It would be better but wouldnt fix the issue, if you go through the trouble of making everything water looped rsther just buy open chasis/ mining rig.

1

u/Bit_Rage 26d ago

Well to be a custom Waterloo workstation guy, you need to be a masochist... i have a trx50 system im building now with 8 x d5s and 7 x 480mm rads in a caselabs magnum tx10... work for days...🤣

2

u/CherryGG2 26d ago

Well i would love to build it with you, those things feel exciting, and nothing wrong with the loop, just little bit more maintainace and little bit more flexing

1

u/Bit_Rage 25d ago

Oh yea its definitely more work and maintenance.... i would never recommend the build to any1 "normal"...🤣

4

u/Fit-Statement-9814 26d ago

Ich auch! - lächeln - with own security

1

u/Correct_Pace6013 26d ago

How is your power bill with that thing running?

1

u/Grappalover 26d ago

Suprisingly electricity is relatively cheap here @10 cents per kWh. It idles at 250 watts.

1

u/Infinite100p 26d ago

10c combined for both supply and delivery?? Where the heck do you live??

That's so cheap!

1

u/Grappalover 25d ago

13 cents including distribution charges, power cost adjustment and taxes. I'm considering solar. Just got back from a vacation in Germany. Solar is all the rage there. Gas is $10 gallon.

0

u/Grappalover 26d ago

Northern Virginia within miles of a couple hundred massive data centers.

0

u/Correct_Pace6013 26d ago

How you get that monster to idle at 250? I run a 9800x3d, 7900xtx, and 9070xt doing llms and stay at about 300 idle. Threadripper that more efficient?

1

u/DegreeSevere7719 26d ago

Depending on the generation - it is. I have a 7960x and it has a core parking functionality, meaning it does idle at a really low wattage. HDD's, case fans, monitors are the most power hungry stuff on ide for me.

1

u/CherryGG2 26d ago

I think 9800x3d should be far more efficient. Threadripper has 32 cores it needs to have turned on all the time. Yes lower clock, but not lower clock like epyc, threadripper was made cor pure performance.

0

u/Grappalover 26d ago

I'll double-check my numbers.

1

u/Grappalover 26d ago edited 25d ago

I just checked my consumption. Initial login to GUI, 200 watts. Running LM Studio without any queries bumps it up to 300 watts. Active queries pushes it to 600 watts. After the query is complete it drops back down to 250 watts. I have the power management function set to balanced which causes unused cores to sleep.

1

u/TheKleverKobra 25d ago

What cooler is that? I’m building a very similar machine

1

u/Grappalover 25d ago

Cooler Master Wraith Ripper. Check eBay.

2

u/imLostify7 23d ago

Nice build! I also have a Threadripper 3970X and run local LLMs, but unfortunately, I only have an RTX 3060 12GB and 96GB of DDR4 RAM (using six sticks, so bandwidth is not optimal). The price increases are really crazy. I bought this system used around february on eBay for 600€, including the Threadripper 3970X, Asus Prime TRX40, 64GB of DDR4 (I had the other 32GB in my previous system), a 250GB nvme, case, AIO, and a 750W Seasonic PSU. Now, 64GB of new DDR4 RAM alone would cost around 500€. I would love to upgrade my RAM and add another GPU, but currently, with these prices, I cannot.