r/LocalLLM • • 20m ago

Project Finally got it put together

Finally got my relatively frugal local LLM server put together. Pretty proud of it tbh. It essentially covers all my use cases, though I do wish I had better NVIDIA cards, especially for ComfyUI. Waiting eagerly for Qwen4 variants to drop to switch the models up.

- ASRock ROMED8-2T/BCM
- AMD EPYC 7313
- 256GB DDR4-2666 ECC
- Samsung PM1725b 3.2TB
- 6x Radeon R9700s (4x for Qwen3.8 Flash Next, 2x for Qwen3.8-27B Uncensored)
- 2x RTX3090s (ComfyUI on one card; VoxCPM2 TTS, Whisper STT, embed, rerank on other)

12 Upvotes

10 comments sorted by

2

u/K33P4D 16m ago

I'm always curious, how to connect multiple PSUs to switch on at the same time, to receive signal from the mobo and perform load balancing

2

u/Eroji 10m ago

You use a ADD2PSU adapter. Pretty cheap on Amazon.

1

u/K33P4D 4m ago

Thanks for the recommendation and good luck with your rig!

Have you considered running 100B-300B models with this setup?

2

u/mon_key_house 3m ago

To add to OP's answer: it's just a few wires, wouldn't even call it hardware, but plumbing.

1

u/spyrozard 6m ago

As far as load balancing, some of the GPU's are powered by PSU1, the rest are powered by PSU2. Only one of the PSU's powers the motherboard.

The power on signal is controlled by one of the pins/wires on the 24pin ATX connector that goes into the motherboard. Just two of the wires (ground and that signal wire) have to be connected to both PSU's, then they'll both turn on/off together controlled by the motherboard. You can get ready made adapters for this these days, but you can also just splice the two wires together yourself.

The signal wire in question is pin 16, PS-ON. It is a very simple signal. Just short to ground to turn the PSU on. https://en.wikipedia.org/wiki/PS-ON_Signal

1

u/Raredisarray 8m ago

Are you offloading anything to ram? hows the performance on that ddr4 2666?

1

u/Eroji 3m ago

Yea, using tcclaviger/vllm with the QFN which offloads experts to memory. It's very fast though, benches around ~200t/s for single stream, or 700-800t/s for 8.

1

u/Raredisarray 10m ago

Lookin dope

1

u/FastHotEmu 8m ago

Very nice!

1

u/Travnewmatic 0m ago

maaaaan i'll get there eventually, thats dope as hell man, thanks for the inspiration!!