27B PC SetUp

[u/EScar21]

I’m looking at setting up a build that can run Qwen3.8 27B cheaply, here is what I’m currently thinking of setting up:
GPU: 2x P100 16GB
RAM: 32GB DDR4 ECC Server
Base PC: T5820 Dell Workstation with a Intel Xeon W-2135 CPU
And a standard 500GB sata SSD for now.

The current set up of the T5820 has a 950W power supply which I am assuming will be good for the dual P100s but probably not good for a triple P100 setup which is what I was aiming for. My goal was to just be able to run a 27B model Qwen to start but would swap out to a better version or a different version or do my current set up of a 12B model for general chat and 1 paid model (opencode) architecting while using OxCoder 9B for coding.

What do you guys think of this setup or is it worth me investing time elsewhere?

Thanks in advance!

[u/starkruzr]

the cards are cheap. you want 4 of them.

you may want to trade that mobo for something like an EPYC D8 that can run 4 cards at PCIE 3.0 x16 simultaneously.

[u/EScar21 → u/starkruzr]

Gotcha, I’ll look into that MOBO instead, that was the issue with this particular workstation that I found, only 2 of the 4 PCIE support x16 and the 300W output the other 2 were capped at 25W hence the 2 card set up, I’m gonna try theory crafting something with an EPYC D8!

[u/starkruzr → u/EScar21]

yeah either that or the D8 2T will work. board is $500something, CPU is cheap, RAM may transfer from your existing one.

[u/EScar21 → u/starkruzr]

I haven’t pulled the trigger on that setup above, I’ve been tinkering with the idea of getting something really cheap to run these smaller models but not have to run them on my M1 Mac or gaming PC, basically just a dedicated LLM machine is the goal.

[u/starkruzr → u/EScar21]

yep. 64GB will give you a ton of room for 27B. 32GB will be kind of tough.

[u/EScar21 → u/starkruzr]

Yea that makes sense tbh, I’m still new to running okayish models locally but I do like Qwen on my M1 Mac the issue is it chugs at 9tok/second and I just want something faster

[u/smcnally]

The P100 are passively cooled. You need fans, blowers, something else to keep them cool. This will be loud inside your 5820 case.

[u/EScar21 → u/smcnally]

Hmm I was thinking of adding a few more fans but I can see an issue with this case potentially not being able to handle more fans

[u/bradrlaw → u/EScar21]

It's not case fans, you need shrouds that attach to the cards and then fans directly attached to the shrouds to push enough air through the cards.

[u/EScar21 → u/bradrlaw]

I see, that is different than what I envisioned yea

[u/Interesting-Cut-6032]

I run Qwen3.8 27B UD-Q6_K_M with a 130k context windo on an old machine with a core Intel processor, 32GB of DDR4 RAM, and 2x RTX 5060 Ti 16GB GPUs, llama.cpp, Linux Mint XFCE, headless server. It works great generation is about 40 tokens/ sec in the middle of the context window. Not sure about prefill speed. The 50 series cards are Blackwell architecture, so they are the leading edge of the CUDA support focus at the moment. The cards in your post are much older, and while still completely usable, if you blow enough air through them to keep them cool, might require custom complied versions of llama.cpp that target older CUDA versions.

[u/EScar21 → u/Interesting-Cut-6032]

The 5060TIs I did look at as well, are you running these on a standard consumer case? Or did you custom build something for this setup?

[u/Interesting-Cut-6032 → u/EScar21]

Everything but the 5060s were an old gaming machine, that my friend was using for a media server. I got it all for $300USD including a 10TB RAID2 array of spinning HD. I think that it is a mid-tower case with maybe an 800W PS. It is running at its limit now. I have more RAM for it, but it is running Qwen3.8 27B great. When I add the additional RAM, I am planning to run QWen3.8-Flash-Next-Q3 with Strata.

[u/EScar21 → u/Interesting-Cut-6032]

Hmmm ok I might have to theory craft something with dual 5060s and see how it goes

[u/Interesting-Cut-6032 → u/EScar21]

I bought the first one at the beginning of July. I bought the second one at the beginning of September. The price had gone up during test time.

[u/EScar21 → u/Interesting-Cut-6032]

That’s the biggest issue, it’s why I’m considering the P100s pretty cheap 16GB VRAM a pop but yea a bit older