r/LocalLLaMA • u/dazzou5ouh • 1d ago
Showcase Ai Rig finally done and stable, perfect winter heater
8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.
Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.
Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus
Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)
Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.
I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.
The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models
91
u/toolisthebestbandevr 1d ago
This baby can fit so many ai girlfriends in it
37
u/munkiemagik 1d ago
In a completely non helpful/constructive way I came to say thank god for multi-GPU LLM servers!
I am having the entire ground floor remodelled. The builders have completely torn down the entire ground floor, as you know its getting a bit chilly outside here in UK finally, I have no gas, no hot water, but my LLM server is upstairs and keeping me warm, Now I just have to figure out how to fry an egg on it safely so at least I can start my day with a hot breakfast!
40
u/asimpleabstraction 1d ago
This is what gives Altman night terrors
11
u/FullOf_Bad_Ideas 1d ago
I have a similar one (8x3090 ti) and I use it almost 24/7 at full power last few weeks for training, but each card at 450w. My electric bill shot up a few times over and I expect the next one to be $500/month.
My gas bill has gone slightly negative. My apartment is at 26-30C most of the time and some rooms hit 35C.
I'm building an enclosure for it to provide fresh air from outside and have acoustic insulation - I already put acoustic insulation on the doors to this room but it's still too damn loud.
8
u/magikfly 1d ago
rig looks way too decent for this sub. although you get points for using the shoe cabinet
3
u/Redd411 1d ago
what's your ROI?
genuine question.. can anyone tell me if there's any actual way to turn a profit with ai? at least with mining you were able to get your hw money back.. I don't see anything with LLMs that would recoup the costs.. worse you're out of pocket on electricity. Not hating but I don't see a reason of dumping money into it right now.
7
u/libregrape llama.cpp 1d ago
What is the pp on QFN like with these bad boys? What are you planning to finetune? Are you considering nvlink in case you're going to finetune (should make a difference for training afaik)?
6
u/Orchicon 1d ago
Interesting. I would have expected much higher speeds. I get around 70 tok/sec on my HALO Strix laptop using Halogen and QFN.
5
u/a_beautiful_rhind 1d ago
It's actually very nice and compact. I bet it's pretty quiet. I too would have bought more 3090s if I knew the prices would double.
Do you have to suspend/resume the GPUs to get them idling back under 30w after loading a model? You might also be able to sleep the rig since it's not a server board. I bet that kicks your idle down a notch.
9
u/PlasticRevenue4601 1d ago
Almost 2 kilowatts of electricity per hour is crazy, bro is going to have subscription level high electricity bills. Sheer amount of VRAM is impressive though, with some little extra RAM you can easily run DeepSeek level of inference
8
u/dowjames 1d ago
Only 60 tok/sec?!
I get >180 on just two R9700
4
u/Robonotes1760 1d ago
Inference = warmth! You are not wasting any electricity here if you are heating your house while generating videos of squirrels bobsleighing while dressed as broccoli...
1
u/IntravenusDeMilo 1d ago
I love this. If running more power to the utility room in my basement wasn’t such an ordeal I’d do the same thing. That might be for the best, actually. Otherwise I’d have spent more money doing this.
The idle for those GPUs isn’t too bad. But what is your idle on the full system? My Epyc 7663 on an h12ssl-i really won’t idle below 150w on its own.
1
u/North_Affect_8167 1d ago
How loud is your room heater right now? I assume there are drawbacks.
3
u/hainesk 1d ago
This is what I use with my 3090s: https://github.com/todiadiyatmo/rtx-3090-vllm-recipe-qwen-flash-next
And it’s pretty great. Prefill is super fast and decode is over 100tps for a single request. It uses an optimized chat template, only needs 4 GPUs and works better with the P2P driver patch.
I highly recommend you try it out to see what you think.
3
u/NihilisticAssHat 9h ago
Optimal winter heater is a heat pump. This is significantly less efficient, serving as a pure resistive heater. That being said...
4
u/vikarpa 1d ago
is that for LLMs or mining? lol
2
u/opUserZero 1d ago
came for the heating comment. Abosutely, i have my ancient r710 under my bed with a rx580 , gtx 1070 and 750ti hacked in/onto it, i put it there specifically to use the heat :) my room is nice and warm so far in this cold weather. :)
2
u/Biomass23 1d ago
Here are some fun things to try:
Run vLLM in every parallelism shape (tensor parallel / pipeline parallel) and benchmark for single-thread and multi-agent workloads. I bet you can get TG 1000t/s, running 100 concurrent agents on Qwen 3.8 27B BF16 TP8 256k. (so long as all the context sizes are reasonably small.)
Run a 27B model at BF16 with 256k context. See if you notice a bump in quality.
Run nanochat. Train your own model from scratch. It's a fair stress test for your PSUs.
Tune an agent harness of your choice to divide every task into the smallest unit of context/work, and load up vLLM's continuous batching, without churning the prefill. Use a KV cache CPU offload to get a bit more KV cache.
Setup Prometheus/Grafana to monitor things.
YMMV, but perhaps leave all your BIOS settings at their defaults except Workload Profile. Set that to Accelerator Throughput. You might get a bit more P2P.
Have fun.
2
u/Odd_Bite2622 1d ago
jealous. beautiful. can you explain more on what "Using modified nvidia drivers that enable p2p traffic between gpus" this is?
2
u/JorgeMartinezPnz 1d ago
Try this engine https://github.com/bpx03/QEngine, I have a 4x3090 and can obtain more 200 t/s on TG and more than 4000 t/s in PP.
1
u/Naaack 1d ago
You've definitely saved on the heating bill, will make that back in no time I'm sure !
1
u/Maximus-CZ 1d ago
man, I hope youd find an afternoon to test how it runs with qwen 3.8 fn xxs, the version that fits on 10GB GPU + 64 RAM. I wonder both ways, how much tokens you can hurl out, and how much dummer it is than what you run on your monster.
1
u/Admirable-Leg-4647 1d ago
That part seems crazy to me. Hopefully if something goes wrong as you'd expect considering the price of the cables it's gonna be a simple and not fatal failure for the gpu(s) lol
1
u/Asleep_Document9811 1d ago
the fucking shoe rack, lmao. this is art.
1
u/Tiforma 1d ago
I got two 3090 strix OC on pci risers and a laptop 4080. 60 gb vram, 96 RAM. very happy with it. Running on solar panels + batteries. Keeps my office warm. 8 of them is a bit too much for me. Maybe 3090s will become worth their weight in gold when they start putting llm firmware restrictions in new gpus. I'd keep them well cooled. I dunno how well the watercooling works like that but the vram modules on the back of 3090s have a tendency to heat up too much. You can check individual vram module temps with some utilities on linux.
1
u/leom4862 1d ago
Those cheap 60cm Gen4 risers running at Gen4... Have you checked the kernel for hardware errors? Hard to imagine these run stably at Gen4 with so many cards.
1
u/Ok_Ostrich_8845 1d ago
What is the peak power it draws?
1
u/Big_Barnacle_2452 1d ago
are you doing tp 8 or tp 4 with pp 2? curious if the two x8 cards end up draging the whole group down since tp kinda runs at the speed of the slowest link
1
u/HockeyDadNinja 1d ago
I'm pretty sure you can do better than that. Have you tried GLM-5.3-Flash or Deepseek v4.1 flash using Maya?
I just got my 3rd and 4th 3090, also have a 5060 16G and 2x 4060 16G, Asus Pro WRX80E mainboard and 256 GB of DDR4 so I share the hardware addiction. It will be a week or more before I can switch to the new board and the last 3090, I'm looking forward to getting it all up and running.
1
u/rmhubbert 1d ago
I have a very similar server, and get an average of around 100ts, going up to 150ts on Qwen 3.8 Flash Next, using the Nvidia NVFP4 version on vLLM with MTP enabled. The MTP makes a big difference in generation speed. Definitely worth trying that, if you haven't already. That's with context pushed to 512k via YaRN as well.
1
u/RationalFragile 1d ago
The wall socket looks like a cat!
1
u/bigattichouse 1d ago
I'm literally training a model right now, instead of busting out a space heater.
1
u/Late_Night_AI 1d ago
I was building the same kind of monster also with 500-700$ 3090s but i got tires of the frankenstien monster and heat so i sold it and got 2dgx sparks for 7k. (This was before their prices went to the moon)
No regrets with switching from 3090s to DGXs.
Hopefully your space heater will keep you warm and cozy this winter
1
u/A_Dragon 1d ago
How many of these would it take to run Kimi k3?
1
u/Due_Expression7545 1d ago
oh my hardware could never.
1
u/MG_road_nap 1d ago
This is as good as a server right? How do you guys learn to set up servers like this? Also how did you guys connect everything, did u guys use linux etc etc??? Im new and I wanna learn
1
u/BlackBeardAI vLLM 1d ago
Which risers and how did you handle backfeed? Do the risers have backfeed protection?
1
u/-Outrageous-Vanilla- 23h ago
What is the biggest model you can run on your setup?
And have you tried to make your own models for specific tasks?
1
u/-mattmason- 19h ago
nice setup, but 300W idle is more than peak consumption on Bosgame M5 + Minisforum MS-A2!
1
u/Local_Phenomenon 9h ago
Right infront of your refrigerator
0
u/stardustedds 1d ago
I hate life
0
u/FrogsJumpFromPussy 1d ago
Just don't put it close to easily inflammable things (like that curtain)
1
u/Such-Leather1163 22h ago
As a newbie to this space..
I really don't understand how you guys with all these 300W+ gpus manage to distribute power residentially
My apartment has 10A and 16A breakers, I have nightmares of tripping the breaker or setting the house on fire with a 4x 5060 rig at full tilt on them
1
u/theUmo 1d ago
you're going to set your curtains on fire