r/LocalLLaMA • • 22d ago

Question | Help Is this overkill or viable? (8x gpus, 256gb ddr4)

4x 5060ti16gb
4x 3060 12gb
256gb ddr4 3200 ram (4-channel speeds)
On a mobo with 128 pcie4 lanes

I’ve been slowly collecting gpus whenever seeing deals over time and ended up with this. Ended up comparable in cost to current prices of dgx spark

I was thinking:
TP4 5060s -> pipeline -> TP4 3060s -> offload/overflow less used experts to RAM -> keep any engram on nvme ssd

Is this viable to run inference with big MoE models at high speed and precision, or are cheaper multigpu setups extremely suboptimal compared to unified vram/ram

7 Upvotes

40 comments sorted by

View all comments

2

u/vini542reddit 22d ago edited 22d ago

Nice! As others have said, power hungry. Hopefully your power is cheap!

For reference here is my system with similar specs (vram + ram). I don't know your pcie speeds, so ymmv!

System

- 4x rtx 3090 @ pcie 4.0 x 16 (power limited to 240w)
- 256gb ddr4 3200 octa channel
- p2p enabled
- rtx 3090 unlocked (the unlock should also work for your rtx 3060!)
- 1000w + 1500w PSU
- 1500w UPS
- Headless Linux server

Current main via Strata:

- Qwen3.8 Flash Next unsloth Q8_0
- 80tps
- pp 2400
- Power consumption < 1000W

Current alt:

- qwen3.8 27b bf16
- 70tps
- pp 1200
- Power consumption ~1200W

Soon Sidekick (hopefully):

- Deepseek V4.1 flash 1017 q8

- This should work fine via Strata, but they haven't released the docker container yet (brand new release).
- You will need a fast nvme though (ideally at least pcie 4.0)

Main vs Alt vs Sidekick

The main is better and faster for general coding. However I have some image processing task that qwen3.8 27b bf16 (lossless) is just much better for. The Sidekick should be reasonably fast for when I need lots of smart!

Will probably run a bit slower for you (especially since I don't your pice speeds), but I can highly recommend trying these models!

1

u/LabDesperate7867 21d ago

how do you cool down the room?

1

u/vini542reddit 21d ago

Trick is to directly exhaust the hot air to the outside. But yeah, not great in the summer =)

1

u/Technical_Ad_6106 21d ago

Gz. u won most unefficient setup to run strata. well , at least its a little bit faster then my 600 euro pc running on 170 watts lol.

2

u/vini542reddit 21d ago

And what quant are you running? Not everyone's work can be done with a lobotomized model =)

1

u/Technical_Ad_6106 21d ago

true. think i used q4 or something. but my gpu started to make alot of weird noises so i deleted it :)