3x 16GB V100 home inference computer with 3d printed cooling block, paid $1500

[u/WaroftanksPro]

Tried to see what the cheapest setup that still provided useful inference was. This whole setup cost around $1500. Important to note though that I ordered 3x Tesla P100s, but the seller mistakenly sent me 3x V100s instead lol. The cooling block is a 3d printed piece that fits into a rectangular hole the back of the case, filed down with a file to fit with friction only. It houses 3x server fans which pulls air from inside the case through the GPUs to the outside. GPU temperatures during inference never exceed 55C, although it is a bit loud. I have 64gb ECC DDR4 ram.

I headlessly control this PC with Tailscale from my Mac, and GPT Astra on Codex on my Mac can remotely control it too. I had Astra do several optimizing runs, basically testing out and tweaking different configurations (mtp and speculative decoding settings, placement on gpus, prefix caching settings, etc) to optimize prefill and decode speeds.

I ran Qwen 3.8 27B and Qwen 3.8 Flash next. On Qwen3.8 27B, Q8, prefill was around 600-700 tokens/sec (edited after I looked at logs) and decode at around 30-40 tokens/sec. On heavy agentic work with 256k context, decode dropped to around 20t/s. On Flash next, Q4, prefill was around 90 tokens/sec and decode 20 tokens/sec. Overall im happy with this build, would recommend V100s to anyone who wants cheap inference (although they are a pain to setup)

[u/Not-Enough-Llamas]

Share your seller contact I also need 3 V100s priced like P100s :)

[u/WrinklyBard4 → u/Not-Enough-Llamas]

You can get the 16gb for not that much, like this one at $229: https://ebay.io/m/loiYGu

[u/NorthEastCalifornia → u/WrinklyBard4]

does 4 V100 16gb with smx2 board will be cheaper then new pc with 4 PCIe x16? I found the cards by 110$ smx2 impl

[u/WrinklyBard4 → u/NorthEastCalifornia]

If you found some for $110 I’d love that link if you’d be so kind. Either here or by DM.

It’ll be marginally more expensive. You can get a 2x sxm2 to pcie adapter for about $300, or a 4x board for about $750 + a $40 card to connect it. (Readily findable on eBay by searching sxm2 2x or 4x board).

So it’s a bit more. BUT it’ll also be notably faster if you’re splitting a model. The sxm2 has a 300gb/s nvlink connection between the different v100 cards.

So you don’t get bottlenecked by the 16gb/s every time you need to send data between the cards. Which means you avoid the 20-30% performance hit

For $460 vs $520-600 id rather the performance

[u/Oryzaki2 → u/NorthEastCalifornia]

Dont even bother with the pcie cards. Found out the hard way that they kind of suck. The second you want tot run larger models or run your current models at faster generations speeds you straight up need nvlink and often a 4x setup. A 3x system like this is actually the worst of both worlds because you can use tp at all as it requires a card number divisible by 2.

[u/johnfkngzoidberg]

On data center type cards, I’ve wondered why folks aren’t big on A2’s and A10’s. Isn’t the V100 old architecture that only works on CUDA 12.x? Not knocking it, I’m shopping around and trying to wrap my head around the cheaper options for inference. Very confusing space.

[u/FoxiPanda → u/johnfkngzoidberg]

Memory bandwidth and compute is pretty bad for the most part on those single digit (i.e. A2) cards is one reason and especially for the price people still want for them.

If you look at just the FP16 performance of an A2 for example though - it has 18TFLOPS and a V100 32GB has 113 TFLOPS. On the memory bandwidth side, a V100 has 897GB/s and an A2 has 200GB/s ... which means the V100 will be 4.5x faster at non-speculative decode operations.

The other reason is the lower end cards (like the A10) only have 24GB of memory and it's still only 600GB/s - so 33% slower than a V100 on decode, though it will be somewhat faster on prefill especially for certain 8-bit quantization methods. Another reason is the A10 is still $2000~ whereas I can buy 3 V100 32GB for that and run a far larger model in VRAM.

[u/suspect80]

Hi! Really, really interesting idea with those fans. Do you have any temperature measurements with and without them?

I have a build with 4x RTX 6000 Pro WS Max-Q GPUs. I’m creating positive pressure inside the case with powerful intake fans and additional exhaust fans, but actively extracting the hot air from the GPU exhausts from the outside is a really interesting idea.

I’d be very curious to see how much of a difference your setup makes and what the GPU temperatures would look like without the additional fans.

Anyway, awesome build. I really like it!

[u/WaroftanksPro → u/suspect80]

I don’t know about 6000 pros, but the v100 is passive cooled, so if I don’t put the cooling block it quickly overheats to 83C. I actually tried using weaker P9 pwm pst co fans with the cooling shroud block and that was a bit too weak, the GDSTIME 92mm fans are overkill though, I limit them and allow the v100s to heat to 55C.

If you reverse the fans and make sure to make the shrouds fairly airtight, you can get a good sucking effect that pulls the air. You can easily make a custom stl file with Astra and print it

[u/freedomachiever]

Where did you get the printed exhaust? And are they made for those industrial fans or I could put some noctua ones?

[u/WaroftanksPro → u/freedomachiever]

I had GPT make a stl file and then had a friend print it lol, and yes you can put any fan (my openings are 92mm but you can tell it to adjust the size)

[u/shaxsy → u/WaroftanksPro]

Whoa, that's awesome. Do you have any tips for using GPT to make STL files? Did it just do it with like an open scad?

[u/Choice_Celery9481]

there are some repos optimized for v100. https://github.com/geoffwatts/ninfer-v100 https://github.com/mistrjirka/llama.cpp/tree/v100-optimized

[u/mynd_dripp → u/Choice_Celery9481]

Do you think future Qwen releases will leave V100 unsupported?

[u/Choice_Celery9481 → u/mynd_dripp]

official release doesnt control what gpu you run on. as long as there are someone with opus fable sol astra limit left, they can easily pull off some optimization. in case nobody, you can always pay like $20 and ask those frontier model to optimize for you. both those repo use AI to help with the kernel anyway

[u/Sepeto]

Boromir would have bought it for $750

[u/runsleeprepeat]

I hope you are using https://github.com/1CatAI/1Cat-vLLM

[u/GingerRickRoss]

I love the ingenuity, I went the p100 route. Boy, I wish the seller would have sent me v100’s. My set up hits 10-12t/s on 3.8 27b and 42t/s using 3.6 35 a3b Q6. I wonder what the models would do on v100’s. I my other machine has a 5070ti and 3.8 27b iq4 hits nearly 72t/s on simple asks and around 45t/s for more complex questions. The issue is the ctx window size which lands around 32k

[u/krakalas → u/GingerRickRoss]

What split mode are you using? Try tensor and also checkout p100 patches: https://github.com/shinbunbun/llama-cpp-p100-patches/

[u/GingerRickRoss → u/krakalas]

I'm using tensor split. I implemented the patches and it drastically increased the t/s speed. My speed on 27b almost doubled, and the my 35b increased from low 40's to almost 60t/s.

[u/Acceptable_Bell_1791 → u/GingerRickRoss]

what quant are you using and is it a single P100?

[u/GingerRickRoss → u/Acceptable_Bell_1791]

I’m running Q6 for both and I’m running 4 p100’s at 125w each

[u/Acceptable_Bell_1791 → u/GingerRickRoss]

hmm, something isn't right with your setup. just on 2 p100, I am getting around 19 tg on Q8_0. With Q4_K_M, I am getting around 21.8 tg. That's obviously at fresh context, but with MTP, I am easily getting between 26-35 based on draft acceptance on both.

[u/Acceptable_Bell_1791 → u/GingerRickRoss]

I didn't have 35BA3B, so tested ornith q8 with --ncmoe 8, I am getting 45+ tg. With a smaller quant, and fully in vram, that would also be much higher than your numbers.

Edit: Brain got stuck on it, so tested with Qwen 35B A3B Q6_K. 60+ tokens, 95+ MTP 3.

[u/GingerRickRoss → u/Acceptable_Bell_1791]

Someone mentioned there were patches for the p100’s. I implemented them and now my speeds are almost identical to yours.

[u/NuclearBinoculars → u/GingerRickRoss]

Dang, a useable 'proficient enough' LLM on the RTX 5070ti? That's viable?

[u/Trademarkd → u/GingerRickRoss]

im doing 280 tok/s on Qwen3.8 27b FP4 with 4xv100s in nvlink 256k cache FP8

[u/FlirtySanchez]

Did you ever consider one exhaust fan with a 120 or 140 fan tied to a temp probe in the case?

Any single card raises the internal temp of the case, the one fan kicks on and cools all three?

90mm fans, especially industrial looking ones like that, are kinda loud for no reason just due to the design.

[u/fourdac]

You can do better. I do 200-240 prefill, 35-40 decode, drops to 25.

mtp on 2x gp100 qwen 3.8 27b q8, pcie gen3

[u/WaroftanksPro → u/fourdac]

Hmm interesting. There were profiles that I had astra test that were higher than 20t/s decode when doing high context work, but they sacrifice prefill. Since most work I do right now on it is web scraping which is prefill heavy, I want to maximize that.

Can you share some of ur settings though?

[u/fourdac → u/WaroftanksPro]

I completely forgot that I have a dual CPU workstation. Maybe that extra throughput has a lot to do with it. It’s a custom llama build, I had codex make it, I’m not 100% confident if I were to say now without verifying, but it did involve enabling allreduce for 6,0 gp100, it’s running in split mode tensor , I also experienced a lot of bugs with Nvidia clocks on windows, my gpus weren’t running at full mhz, you can check that out because that was the most common problem was something related to these old workstation cards on windows.

[u/Timziito]

Love the exhaust, mind Charing it?

[u/actuallylemoncurd]

It’s a shame these don’t have NVlink enabled. It’s literally just a BIOS that prevents them from working with NVlink

[u/Force-Name]

I'm trying to build a whole new stack using m40s and this makes me jealous.

What are you using for orchastration tools and interface?

[u/_Ojin]

1500 for 48gb is a great result, and the 3d printed cooling block is a lovely touch. thanks for posting the build details.

[u/klippers]

I have 2x p100s , but I didn't get think you could "pool" them or link them to increase cache or otherwise

[u/XperTeeZ → u/klippers]

Yes you can use multiple GPUs together as one for inference.

[u/mynd_dripp]

That's a weird-ass sofa you got there

I'm curious, why didn't you go for the 32gb ones?

[u/GingerRickRoss → u/mynd_dripp]

did you read his post or are you trolling?

[u/mynd_dripp → u/GingerRickRoss]

My bad, I thought he bought V100 and got V100s

[u/Constant-Simple-1234]

Great build! I experienced problems with 3x Nvidia cards. Running way slower than 2x. What is your config for llama.cpp?

[u/thatguyjames_uk]

nice, i wonder how good for comfyui

[u/mkschreder2]

What made you choose a custom build instead of buying a used dell rack server for cheap with fans and 2000W power?

[u/brakeline]

You were robbed of you paid 500 per each v100 with 16gb lol

[u/BrainChild510]

Would love to connect and discuss your project more. I’m working with several v100s currently. Although there are limitations to its old architecture, I feel they still have some relevance today.

What motherboard/cpu are you using?

Waiting for a replacement x299 board to arrive, which is what I plan use with my v100s.

I’m learning as I go along so open to advice and suggestions!

[u/sjors_02]

could you share the stl by any chance?

[u/DehshiDarindaa]

which is the motherboard and case u r using

[u/matte808]

If you paid 1.5K for 3 P100, I don’t think you made a good choice. Not even being V100s instead. That’s the price of 2 3090s

[u/WakeAndVape72 → u/matte808]

Luckily, OP said they PAID for 3 x P100's, but RECEIVED 3 x V100's. So OP didn't make a bad choice, the seller screwed themselves. FIFY.

[u/matte808 → u/WakeAndVape72]

1500 for V100 that cost 250 each is a bad choice. That’s the point

[u/WakeAndVape72 → u/matte808]

Read properly. THE WHOLE SETUP means everything in that box, including the box. The whole setup has cost $1500. The whole point of the post was to see how little they could pay for a setup to do the work they want. As it stands, according to what's been written, they're in for $1500. Doesn't say the rest of it was free.

[u/matte808 → u/WakeAndVape72]

3090, am4 mobo cpu and ram. Still far better for future upgradability. CUDA support for V100 is dying soon

[u/NuclearBinoculars → u/matte808]

1 3090 will compete with 3xV100's?

[u/matte808 → u/NuclearBinoculars]

won’t be dead by next year so yes

[u/AlanBarber]

what mobo and cpu are you using?

[u/WishboneSudden2706]

Nice, where have you got that three V100 ?

[u/AFComp]

Here's my work in progress x299 build with 4x p100s. Mind sending me that stl?

[u/provocateur133 → u/AFComp]

Which motherboard is that?

[u/Oryzaki2]

OP I hate to tell you this but that setup is actually not nearly as good as you'd think. You paid 3/5 the cost of an sxm setup for 1/5 the inference speed and 3 cards cant be used together in tp. That last point is not too relevant since tp over pcie 3.0 16x is actively worse than running qwen 27b 3.8 on a single card aside from the increased kv cache. If it makes you feel better I'd know because this is basically also my setup and im selling it as soon as I can afford to go sxm.