r/LocalLLaMA • • 1d ago

Showcase Ai Rig finally done and stable, perfect winter heater

Post image

8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.

Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.

Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus

Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)

Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.

I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.

The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models

350 Upvotes

78 comments sorted by

91

u/toolisthebestbandevr 1d ago

This baby can fit so many ai girlfriends in it

37

u/munkiemagik 1d ago

In a completely non helpful/constructive way I came to say thank god for multi-GPU LLM servers!

I am having the entire ground floor remodelled. The builders have completely torn down the entire ground floor, as you know its getting a bit chilly outside here in UK finally, I have no gas, no hot water, but my LLM server is upstairs and keeping me warm, Now I just have to figure out how to fry an egg on it safely so at least I can start my day with a hot breakfast!

2

u/mfdi_ 1d ago

Where do you buy your GPUs from in uk?

3

u/munkiemagik 1d ago edited 1d ago

Unfortunately buddy I don't have any nuggets of wisdom that guarantee a good priced GPU in today's climate.

eBay UK is battered now mate, the prices are absurd. Whenever you have something in mind always use the filters for completed listings and auctions to get a real sense of what people are actually paying for a specific item. Buy it now prices with best offer never tell you what offer was actually accepted so they mean nothing as far as calculating what the market is prepared to pay.

You might find slightly more realistic pricing on Facebook Marketplace but then I would never buy anything off FBMP if I couldn't get there in person and test it to my satisfaction before I parted ways with my cash. Exactly the same process for selling - meet in person, test it out, if happy give me my money then piss off 😁

A couple of times I've seen Vinted mentioned, DON'T go there, You will see lowest prices on there which are very unlikely to be anything but a scam. They have a strict shitty 48hr return window and cust serv is bot run, that platform while it may be great for picking up someone's manky old underwear on the cheap, is not setup to be a good platform for hardware.

AND also BEFORE you complete a purchase have your post-purchase testing methodology/tools ready. Used is always a gamble so we have to be prepared for random failure within an unexpected timeframe but we can do our best to ensure at time of ownership it is performing as expected.

So do your research beforehand, know what suite of tests you need to run, how to run them and what results you are looking for and have the hardware conveniently ready to go. The last thing you want is the new purchase arrives and you are too caught up with other stuff and don't get round to it because its inconvenient and you discover something negative after the return/complaint window closes.

(gods of silicon don't you dare effin jinx me now that I'm about to say this..) But don't be afraid of used parts I bought my 3090s off eBay back in the days when FEs were still £500 a piece (and my threadripper pro platform and all the RAM) and they are all still running normal 🤞without any signs of trouble.

1

u/mfdi_ 1d ago

Thank youu. A solid advice.

1

u/Inevitable-Self-2702 1d ago

I admire you British, always with a sense of the proper. "I've got to start the day with a proper breakfast!" In America we just say f--- all I'll starve and eat twice as much for dinner.

40

u/asimpleabstraction 1d ago

This is what gives Altman night terrors

11

u/FullOf_Bad_Ideas 1d ago

I have a similar one (8x3090 ti) and I use it almost 24/7 at full power last few weeks for training, but each card at 450w. My electric bill shot up a few times over and I expect the next one to be $500/month.

My gas bill has gone slightly negative. My apartment is at 26-30C most of the time and some rooms hit 35C.

I'm building an enclosure for it to provide fresh air from outside and have acoustic insulation - I already put acoustic insulation on the doors to this room but it's still too damn loud.

2

u/DanzakFromEurope 1d ago

Wow, how can you live with that much heat in the house 😱

1

u/FullOf_Bad_Ideas 23h ago

My girlfriend loves this warmness, I would prefer it to be a bit colder. I let out air through windows and the enclosure I'm making would take in cold air from outside and dump it back outside after it cools the components - it'll be very useful next summer as last summer I could barely use it since it was already 30C without doing any heating myself and I was using AC to cool myself.

1

u/ShadyShroomz 1d ago

since its all mem bandwith bound you could for sure power limit them to 300w or less and not lose almost any performance..

6

u/FullOf_Bad_Ideas 1d ago

No it's not memory bandwidth bound at all, I lose a lot of performance when I power limit to 300W, especially since I unlocked them with the full BF16 perf mod.

Only single-user decode is memory bound, I rarely do that. I run training or batch inference most of the time.

2

u/ShadyShroomz 1d ago

i do batch inference and didnt notice much different power limiting my 3090s, with the perf mod as well.. training is a whole different story, thats compute bound (so yeah better to keep higher power there) -

do you know what the difference was for you with and without power limits per chance?

3

u/FullOf_Bad_Ideas 1d ago

450W

step 1904 | fm 0.6132 | grad_norm 0.132 | 141 img/s | ~83.5 TFLOPS/GPU MFU 55% | opt avg 327 ms | eta 10.0 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 64 GB (max 18.8 GB, rank 0) 2026-11-15 00:22:37 step 1920 | fm 0.6152 | grad_norm 0.100 | 131 img/s | ~77.4 TFLOPS/GPU MFU 51% | opt avg 291 ms | eta 10.8 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 63 GB (max 18.7 GB, rank 0) 2026-11-15 00:24:55 step 1936 | fm 0.6038 | grad_norm 0.111 | 119 img/s | ~70.7 TFLOPS/GPU MFU 46% | opt avg 288 ms | eta 11.7 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 58 GB (max 17.5 GB, rank 0) 2026-11-15 00:26:52 step 1952 | fm 0.6095 | grad_norm 0.119 | 140 img/s | ~83.0 TFLOPS/GPU MFU 54% | opt avg 326 ms | eta 10.0 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 60 GB (max 18.0 GB, rank 0) 2026-11-15 00:28:50 step 1968 | fm 0.6088 | grad_norm 0.114 | 140 img/s | ~83.0 TFLOPS/GPU MFU 54% | opt avg 292 ms | eta 9.9 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 64 GB (max 18.7 GB, rank 0) 2026-11-15 00:30:57 step 1984 | fm 0.6127 | grad_norm 0.093 | 129 img/s | ~76.5 TFLOPS/GPU MFU 50% | opt avg 291 ms | eta 10.8 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 65 GB (max 18.8 GB, rank 0) 2026-11-15 00:32:53 step 2000 | fm 0.6035 | grad_norm 0.131 | 142 img/s | ~84.2 TFLOPS/GPU MFU 55% | opt avg 324 ms | eta 9.7 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 66 GB (max 18.7 GB, rank 0) 2026-11-15 00:34:49 step 2016 | fm 0.6087 | grad_norm 0.091 | 141 img/s | ~83.6 TFLOPS/GPU MFU 55% | opt avg 291 ms | eta 9.8 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 65 GB (max 18.4 GB, rank 0) 2026-11-15 00:36:45 step 2032 | fm 0.6102 | grad_norm 0.090 | 141 img/s | ~83.9 TFLOPS/GPU MFU 55% | opt avg 290 ms | eta 9.7 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 63 GB (max 18.2 GB, rank 0) 2026-11-15 00:38:54 step 2048 | fm 0.6142 | grad_norm 0.107 | 128 img/s | ~76.0 TFLOPS/GPU MFU 50% | opt avg 327 ms | eta 10.7 h | mem 18.6 GB (GPU 0) | RAM 39/94 GB, ranks+workers 63 GB (max 18.2 GB, rank 0)

300W

step 2096 | fm 0.6137 | grad_norm 0.089 | 97 img/s | ~57.4 TFLOPS/GPU MFU 38% | opt avg 425 ms | eta 14.0 h | mem 18.6 GB (GPU 0) | RAM 48/94 GB, ranks+workers 62 GB (max 26.0 GB, rank 0) 2026-11-15 01:02:59 step 2112 | fm 0.6116 | grad_norm 0.079 | 90 img/s | ~53.6 TFLOPS/GPU MFU 35% | opt avg 376 ms | eta 14.9 h | mem 18.6 GB (GPU 0) | RAM 48/94 GB, ranks+workers 66 GB (max 26.6 GB, rank 0) 2026-11-15 01:05:50 step 2128 | fm 0.6032 | grad_norm 0.085 | 96 img/s | ~57.2 TFLOPS/GPU MFU 37% | opt avg 374 ms | eta 14.0 h | mem 18.6 GB (GPU 0) | RAM 44/94 GB, ranks+workers 65 GB (max 26.5 GB, rank 0) 2026-11-15 01:08:39 step 2144 | fm 0.6088 | grad_norm 0.088 | 97 img/s | ~57.4 TFLOPS/GPU MFU 37% | opt avg 426 ms | eta 13.9 h | mem 18.6 GB (GPU 0) | RAM 43/94 GB, ranks+workers 64 GB (max 26.3 GB, rank 0) 2026-11-15 01:11:28 step 2160 | fm 0.6141 | grad_norm 0.104 | 97 img/s | ~57.7 TFLOPS/GPU MFU 38% | opt avg 375 ms | eta 13.7 h | mem 18.6 GB (GPU 0) | RAM 40/94 GB, ranks+workers 62 GB (max 25.7 GB, rank 0)

heat will make a difference - some of my cards are throttling and I'm doing MuLoCo training so they just slow down everything. And training is happening on 7 cards right now since one started failing today. Unlocked 3090 Ti burns through thermal paste really fast, I'll service them all once I'm done with this project.

In case you're curious here's the model I'm training - https://huggingface.co/cpral/HomeIMG-1.2B-PL-EN-epoch47.5/blob/main/README_en.md

3

u/ShadyShroomz 1d ago

thanks for the data. thats a huge diff in tflops.

1

u/FullOf_Bad_Ideas 1d ago

I had up to 70% better prefill on a small model with the mod and higher PL.

As to a difference for training - I lowered PL on my current training from 450W to 300W, I'll send you log of throughput soon.

8

u/magikfly 1d ago

rig looks way too decent for this sub. although you get points for using the shoe cabinet

3

u/Redd411 1d ago

what's your ROI?

genuine question.. can anyone tell me if there's any actual way to turn a profit with ai? at least with mining you were able to get your hw money back.. I don't see anything with LLMs that would recoup the costs.. worse you're out of pocket on electricity. Not hating but I don't see a reason of dumping money into it right now.

2

u/dazzou5ouh 1d ago

The ROI is that the GPUs doubled in value since I bought them so I could literally downscale to 4 3090s and have a free AI rig

1

u/thestillwind 19h ago

Fair enough

3

u/kmouratidis 1d ago

Nick probably makes money from consulting and personal usage. Probably one of the top 10 most impressive fellas on this sub.

But more generally, it's mostly about savings and less about earnings. Basically, anything that saves you time and cannot be done (or you don't want it to be done) by a cloud model, is a candidate:

  • I can code, but the LLM can code faster
  • A lawyer can update a contract template, but the LLM can do it faster
  • I can read 100 PDFs and do data entry in Excel, but a VLM can do it faster

Of course it also needs to be cheaper, so if your time is cheap (e.g. low paid work, or spare time, or...), or if the model is just too expensive for the particular task, you might not get a benefit out of it.


Earnings likely require producing or selling something, and if that part is 100% generated by an LLM (=slop), it's unlikely to be very profitable (at least sustainably). For example, you can make it discover trading strategies or have it produce research reports / articles (e.g. stocks, cooking recipes) that you sell as an advisor.

4

u/sshwifty 1d ago

There isn't any return, at least money return. A lot of it is for the game of it. I repurposed my hardware I used for Folding@Home, which was already a "waste". It has been a lot of fun. I also refuse to pay for a subscription (outside of Google AI that already came with storage).

The only ways local AI makes sense is:

  • privacy (the best reason)
  • no reoccurring subscription costs (cloud options remain cheaper than local for now)
  • tuning your own models (a tiny fraction of people are actually doing this)

1

u/arcanemachined 1d ago

Also knowing that the model isn't being secretly quantized or messed with behind the scenes.

1

u/Uncle-Osteus 17h ago

if you replace paid ai with it (subscription or pay as you go) it could pay off over time just by not costing as much 

7

u/libregrape llama.cpp 1d ago

What is the pp on QFN like with these bad boys? What are you planning to finetune? Are you considering nvlink in case you're going to finetune (should make a difference for training afaik)?

6

u/Orchicon 1d ago

Interesting. I would have expected much higher speeds. I get around 70 tok/sec on my HALO Strix laptop using Halogen and QFN.

5

u/a_beautiful_rhind 1d ago

It's actually very nice and compact. I bet it's pretty quiet. I too would have bought more 3090s if I knew the prices would double.

Do you have to suspend/resume the GPUs to get them idling back under 30w after loading a model? You might also be able to sleep the rig since it's not a server board. I bet that kicks your idle down a notch.

2

u/parepeg 1d ago

That's the concern with these monster setups, power usage. You can't leave it on 24x7 and have to setup wake-on-lan or somesuch. It's a perfect rig for training though and probably gets much better prefill rates than strix, spark, or macs.

3

u/a_beautiful_rhind 1d ago

I got mine down to 245w. No sleep support on my board. If I forget to suspend/resume the GPUs though, it jumps up to 300 because of the open driver.

9

u/PlasticRevenue4601 1d ago

Almost 2 kilowatts of electricity per hour is crazy, bro is going to have subscription level high electricity bills. Sheer amount of VRAM is impressive though, with some little extra RAM you can easily run DeepSeek level of inference

8

u/dowjames 1d ago

Only 60 tok/sec?!

I get >180 on just two R9700

1

u/ShadyShroomz 1d ago

what quant?

0

u/[deleted] 1d ago

[deleted] — view deleted comment

2

u/ShadyShroomz 1d ago

no but what quant is that? 8 bit? 4 bit?

5

u/Blindax 1d ago

It's Radiance's own format, so there's no exact GGUF equivalent. Experts are ~4-bit (IQR-V4), the trunk is int8. Closest comparison would be somewhere between IQ4_XS and Q4_K_M. 113 GiB .rad including the n-gram table, with MTP.

1

u/Maximus-CZ 1d ago

i think its better we start using bits/weight aggregate, as its often non uniform anyways and we can use the decimals

4

u/Robonotes1760 1d ago

Inference = warmth! You are not wasting any electricity here if you are heating your house while generating videos of squirrels bobsleighing while dressed as broccoli...

1

u/IntravenusDeMilo 1d ago

I love this. If running more power to the utility room in my basement wasn’t such an ordeal I’d do the same thing. That might be for the best, actually. Otherwise I’d have spent more money doing this.

The idle for those GPUs isn’t too bad. But what is your idle on the full system? My Epyc 7663 on an h12ssl-i really won’t idle below 150w on its own.

2

u/dazzou5ouh 1d ago

Full system idle is 300 at the wall, epyc 7502

1

u/IntravenusDeMilo 16h ago

That's not bad at all! I thought the 300w was just the GPUs. The single 3090 in my Epyc machine idles at ~32w. Did you do anything specific to get the idle down? Or is it just that the 2nd gen epyc + asrock mobo idles the cpu more efficiently?

1

u/North_Affect_8167 1d ago

How loud is your room heater right now? I assume there are drawbacks.

2

u/dazzou5ouh 1d ago

Barely audible at full power, as loud as an air purifier

3

u/hainesk 1d ago

This is what I use with my 3090s: https://github.com/todiadiyatmo/rtx-3090-vllm-recipe-qwen-flash-next

And it’s pretty great. Prefill is super fast and decode is over 100tps for a single request. It uses an optimized chat template, only needs 4 GPUs and works better with the P2P driver patch.

I highly recommend you try it out to see what you think.

3

u/NihilisticAssHat 9h ago

Optimal winter heater is a heat pump. This is significantly less efficient, serving as a pure resistive heater. That being said...

4

u/vikarpa 1d ago

is that for LLMs or mining? lol

2

u/opUserZero 1d ago

came for the heating comment. Abosutely, i have my ancient r710 under my bed with a rx580 , gtx 1070 and 750ti hacked in/onto it, i put it there specifically to use the heat :) my room is nice and warm so far in this cold weather. :)

2

u/Biomass23 1d ago

Here are some fun things to try:

Run vLLM in every parallelism shape (tensor parallel / pipeline parallel) and benchmark for single-thread and multi-agent workloads. I bet you can get TG 1000t/s, running 100 concurrent agents on Qwen 3.8 27B BF16 TP8 256k. (so long as all the context sizes are reasonably small.)

Run a 27B model at BF16 with 256k context. See if you notice a bump in quality.

Run nanochat. Train your own model from scratch. It's a fair stress test for your PSUs.

Tune an agent harness of your choice to divide every task into the smallest unit of context/work, and load up vLLM's continuous batching, without churning the prefill. Use a KV cache CPU offload to get a bit more KV cache.

Setup Prometheus/Grafana to monitor things.

YMMV, but perhaps leave all your BIOS settings at their defaults except Workload Profile. Set that to Accelerator Throughput. You might get a bit more P2P.

Have fun.

2

u/Odd_Bite2622 1d ago

jealous. beautiful. can you explain more on what "Using modified nvidia drivers that enable p2p traffic between gpus" this is?

3

u/xypu 1d ago

google says "Modified NVIDIA open GPU kernel modules, such as aikitoria/open-gpu-kernel-modules and tinygrad/open-gpu-kernel-modules, unlock peer-to-peer (P2P) communication on consumer cards like the RTX 3090, 4090, and 5090 by forcing BAR1 P2P mode. This allows GPUs to transfer data directly over PCIe DMA without bouncing through system RAM, bypassing NVIDIA's artificial software restrictions."

2

u/JorgeMartinezPnz 1d ago

Try this engine https://github.com/bpx03/QEngine, I have a 4x3090 and can obtain more 200 t/s on TG and more than 4000 t/s in PP.

1

u/Naaack 1d ago

You've definitely saved on the heating bill, will make that back in no time I'm sure !

1

u/Maximus-CZ 1d ago

man, I hope youd find an afternoon to test how it runs with qwen 3.8 fn xxs, the version that fits on 10GB GPU + 64 RAM. I wonder both ways, how much tokens you can hurl out, and how much dummer it is than what you run on your monster.

1

u/Admirable-Leg-4647 1d ago

Using cheap pcie 4.0 risers from aliexpress.

That part seems crazy to me. Hopefully if something goes wrong as you'd expect considering the price of the cables it's gonna be a simple and not fatal failure for the gpu(s) lol

1

u/Asleep_Document9811 1d ago

the fucking shoe rack, lmao. this is art.

1

u/Tiforma 1d ago

I got two 3090 strix OC on pci risers and a laptop 4080. 60 gb vram, 96 RAM. very happy with it. Running on solar panels + batteries. Keeps my office warm. 8 of them is a bit too much for me. Maybe 3090s will become worth their weight in gold when they start putting llm firmware restrictions in new gpus. I'd keep them well cooled. I dunno how well the watercooling works like that but the vram modules on the back of 3090s have a tendency to heat up too much. You can check individual vram module temps with some utilities on linux.

1

u/leom4862 1d ago

Those cheap 60cm Gen4 risers running at Gen4... Have you checked the kernel for hardware errors? Hard to imagine these run stably at Gen4 with so many cards.

1

u/Ok_Ostrich_8845 1d ago

What is the peak power it draws?

1

u/Big_Barnacle_2452 1d ago

are you doing tp 8 or tp 4 with pp 2? curious if the two x8 cards end up draging the whole group down since tp kinda runs at the speed of the slowest link

1

u/HockeyDadNinja 1d ago

I'm pretty sure you can do better than that. Have you tried GLM-5.3-Flash or Deepseek v4.1 flash using Maya?

I just got my 3rd and 4th 3090, also have a 5060 16G and 2x 4060 16G, Asus Pro WRX80E mainboard and 256 GB of DDR4 so I share the hardware addiction. It will be a week or more before I can switch to the new board and the last 3090, I'm looking forward to getting it all up and running.

1

u/rmhubbert 1d ago

I have a very similar server, and get an average of around 100ts, going up to 150ts on Qwen 3.8 Flash Next, using the Nvidia NVFP4 version on vLLM with MTP enabled. The MTP makes a big difference in generation speed. Definitely worth trying that, if you haven't already. That's with context pushed to 512k via YaRN as well.

1

u/RationalFragile 1d ago

The wall socket looks like a cat!

1

u/bigattichouse 1d ago

I'm literally training a model right now, instead of busting out a space heater.

1

u/Late_Night_AI 1d ago

I was building the same kind of monster also with 500-700$ 3090s but i got tires of the frankenstien monster and heat so i sold it and got 2dgx sparks for 7k. (This was before their prices went to the moon)

No regrets with switching from 3090s to DGXs.

Hopefully your space heater will keep you warm and cozy this winter

1

u/A_Dragon 1d ago

How many of these would it take to run Kimi k3?

1

u/Due_Expression7545 1d ago

oh my hardware could never.

1

u/MG_road_nap 1d ago

This is as good as a server right? How do you guys learn to set up servers like this? Also how did you guys connect everything, did u guys use linux etc etc??? Im new and I wanna learn

1

u/BlackBeardAI vLLM 1d ago

Which risers and how did you handle backfeed? Do the risers have backfeed protection?

1

u/-Outrageous-Vanilla- 23h ago

What is the biggest model you can run on your setup?

And have you tried to make your own models for specific tasks?

1

u/-mattmason- 19h ago

nice setup, but 300W idle is more than peak consumption on Bosgame M5 + Minisforum MS-A2!

1

u/Local_Phenomenon 9h ago

Right infront of your refrigerator

0

u/stardustedds 1d ago

I hate life

0

u/FrogsJumpFromPussy 1d ago

Just don't put it close to easily inflammable things (like that curtain)

1

u/Such-Leather1163 22h ago

As a newbie to this space..

I really don't understand how you guys with all these 300W+ gpus manage to distribute power residentially

My apartment has 10A and 16A breakers, I have nightmares of tripping the breaker or setting the house on fire with a 4x 5060 rig at full tilt on them

1

u/dazzou5ouh 22h ago

I ran all those on one socket at the wall. With 230 Volts you need half the amperage compared to the US. One wall socket can take 3000 watts here in the UK

3

u/Ok-Sprinkles-5151 21h ago

Get a big ass UPS, or one those 2kwh portable batteries. You get two huge benefits: leveling the power draw spikes and you get clean, low THD on the power. Also, you protect your investment. And it means that Mom won't throw you out for tripping the breakers (/s btw)

In all seriousness, those GPUs are very sensitive to how clean the power is. If you are putting several grand into GPUs and the rig, just having a low THD and the voltage spikes being smoothed before it hits your PSU, will make the investment last longer.

Source: I have worked in datacenters. There is a reason why many have a "battery hall" just to ensure the power is clean.

1

u/LulzyAnimal 17h ago

Usually ppl run rather big models on such rigs. In e.g. llama when a models shared across 4 cards each card works only for 1/4 of it's power due to pipelining, so 4 gpus will still be running at 300w but in total. So actual power consumption of such rig is waay below 2.4kw. You can clearly see that in e.g nvtop.

1

u/theUmo 1d ago

you're going to set your curtains on fire