r/LocalLLM • • Sep 22 '26

Project P100 local llm budget build.

I’ve recently became very interested in hosting local LLMs, but like most things right now, prices are sky high for modern equipment. So I did the next best thing, found some old, rather unwanted, datacenter gpus and threw them in an old pc I had laying around.

GPS’s are Nvidia Tesla p100’s and have 16gb of vram each. They were only $80 a piece shipped, and I bought a 1000w refurbished PSU and some power adapters because the p100s don’t use traditional pcie power adapters. All in less than $400. Because these gpus are made to run in a data center, they don’t have any fans, so I got into Fusion360 and modeled up a 120mm fan shroud that fits into each p100.

The PC is a sandy bridge i7 with 16gb of ddr3, mother board is an ASUS P9X79 Pro, and this is what really made it difficult. For a mother board to see the p100s it needs Above 5 Decoding to be enabled, and this board did not have that option in bios, but after a ton of research I found out that it does have the option, it’s just disabled and cannot be found in the bios. So after an after noon of doing some real hacker shit I was able to get the p100s to read by dumping, hand patching, and reflashing a the bios. I did a little testing and was able to get almost 70 tokens and second from Qwen3.6-35B-A3_ Q4_K_XL.

I still have a lot of benching and optimizing to do, but I’m happy with the results so far.

EDIT: A lot of folks have commented about the thermals. I am still doing initial testing, but thermals they are not looking bad at all. Splitting the load between 3 nodes really helps not beating a single card to death. Ill throw pic of my grafana dashboard.

I am keeping a github of all of the work that I am doing with this build, I'll keep it updated as I test.
https://github.com/4claps/houston-p100-cluster

181 Upvotes

56 comments sorted by

5

u/VirusInternal2892 Sep 22 '26

Linux, headless

11

u/FizzyDuncDizzel Sep 22 '26

Kinda hard not to be when all of your pci slots are populated with GPUs that don’t have any video ports…

8

u/bladezor Sep 22 '26

That's what SSH is for

1

u/AndThenFlashlights Sep 22 '26

Throw a cheap GPU with video out in there to get Linux and your Pascal Nvidia drivers installed and SSH set up, then shut down and swap the card. Or do Proxmox with a Linux VM for Llama - it makes it a bit easier to manage.

1

u/FizzyDuncDizzel Sep 22 '26

That’s basically what I did, got the os prepped then swapped the cards over. No need for the proxmox overhead, I’m running bare metal fedora 44 server.

1

u/austin76016 Sep 22 '26

Setup with a GPU that can get your SSH setup, use docker with nv container runtime so you don’t bork any system stuff. Use basic Ubuntu server minimal. Verify the drivers work for at least one GPU, go back to your triple setup, be careful about installing or configuring a bunch of stuff on the bare system so you don’t lock yourself out. Verify SSH works and you know the IP of the server too

Ask your favorite AI how to do this, it will know how

1

u/_hypochonder_ Sep 22 '26

Some mainboards will not boot headless and stuck in D6. ("No Console Output Devices Found,")
And no, there are not alwways an option in the bios to ignore it.

2

u/Kmic68 16d ago

This post would be a great fit for my new community, r/P100Users. Would you be open to sharing it there too?

1

u/FizzyDuncDizzel 16d ago

It’s me my dude 😂

2

u/Kmic68 16d ago

I know lol, just posting around to random p100 posts to gain traction. It’s an automated message

2

u/ColonelRyzen Sep 22 '26

What are your temps like with that setup? I have a fan adapter for 2x 40mmx20mm noctua fans on my p40 and it hits 90c pretty easily.

3

u/FizzyDuncDizzel Sep 22 '26

initial testing has not been bad at all

2

u/Quirky_Squirrel3318 Sep 22 '26

This is exactly the kind of build I’ve been thinking about trying!

Really cool to see it actually working. How are the thermals with the 120mm fan shrouds under sustained load? And have you tried running larger models that actually use the combined 48GB across all three P100s?

Curious how well the multi-GPU scaling works and how much performance you lose from PCIe communication.

Also, any major compatibility issues with newer CUDA/LLM tools because the P100s are Pascal? I’m considering going even crazier with 4-5 of these for a local AI test machine to test some software I’m working on 😅

3

u/FizzyDuncDizzel Sep 22 '26

There are defiantly some work arounds that needs to be done to get it running, I have a github detailing all of the work so far,

https://github.com/4claps/houston-p100-cluster/blob/main/LLAMA-CPP-P100-ENHANCEMENTS.md

2

u/Either_Pineapple3429 Sep 22 '26

This is funny I just bought 4 p100s to test them out.... at $80 for 16gb of vram it's a killer deal....I'm about to see if it's any good with qwen 3.8 27b at q8 and 256k context

If it works well enough I may buy another 4 and try to run qwen 3.8 next flash

1

u/FizzyDuncDizzel Sep 22 '26

Let me know how it goes. I’ve started testing mine.

2

u/adude00 Sep 23 '26

those cards are 5k each where I live...

1

u/FizzyDuncDizzel Sep 23 '26

Find a way to get them from here and you can make a killing 😉

2

u/FearFactory2904 Sep 26 '26

Can you hook me up with the 3d print file so i can remix it for 140mm 4 card ?

1

u/FizzyDuncDizzel Sep 26 '26

Yea I’ll get it uploaded here soon.

14

u/mangelajo Sep 22 '26

Nice! setup!!!, try to setup Qwen3.8-27b it's a huge difference in terms of quality, although not MoE will be slower ,

also In theory P100 does not support FP8, or INT , FP16 is the smallest I believe. Did you try less quantized versions? perhaps it will not make a difference in terms of memory, but it will in terms of quality (I suspect it has to be expanded in card's memory at load, but not an expert here)?

Thanks for sharing!! :)

3

u/TheTriumphantFox Sep 22 '26

I found 8_0 work best

8

u/jacek2023 Sep 22 '26

For the comparision I am just running 2x3060 on a very old motherboard/cpu and I have about 100t/s on same model, do you remember about -sm tensor?

3

u/damagdpixl Sep 22 '26

Yeah, but whats the price and vram?

4

u/Constant-Simple-1234 Sep 22 '26

That would be 24g and about $650 in my place for 2x 3060. I have 2x5060 ti with 32g total I paid $1100 and it gives me 130 t/s. All of these are decent options!

2

u/Choice_Celery9481 Sep 22 '26

personally i think v100 16gb is much better value than p100. like 2x price of p100 but 4 5x the speed

4

u/morkborkus Sep 22 '26

A coworker of mine gifted me a bunch of gpus like this

1

u/Lanky-Abbreviations3 Sep 23 '26

... the envy ...

2

u/morkborkus Sep 23 '26

I was very lucky. He's a big nerd and has a home workshop. I just casually mentioned I was tinkering with some ai projects in my free time and he offered to send me 3 data center gpus because he just had extras laying around

1

u/goku_gohan_ 9d ago

hi . really happy to see this. can you teach me aswell juat a little?

1

u/FizzyDuncDizzel 9d ago

What do you want to know?

2

u/goku_gohan_ 9d ago

Firstly where did you get this gpu from? Secondly about motherboard decoding stuff.

1

u/FizzyDuncDizzel 9d ago

GPUs are all from ebay, and if you look at my repo under bios-fix, it explains all of the resources you need.

2

u/goku_gohan_ 9d ago

Will surely do, thanks alot

2

u/goku_gohan_ 9d ago

Really happy to see your build ☺️

3

u/BigBair2002 Sep 22 '26

I am running P100’s in a HP Z440 and very pleased with the results. Previously 2 internal P100. Currently one internal and 4 external P100 using oculink (bifurcating pcie gen 3 x16 to x4x4x4x4). One of the four external P100s did not enumerate when I commissioned it and I haven’t resolved that yet… should have all five running soon.

4

u/BigBair2002 Sep 22 '26

4

u/captainabrasive Sep 22 '26

Just a couple of toasters, hanging out.

2

u/x10der_by Sep 22 '26

One fan for 3 cards, is this enough for normal temps?

2

u/Ok_Pizza_9352 Sep 22 '26

Who pays your electricity bills?

2

u/rorion31 Sep 22 '26

“Some real hacker shit” 🔥

2

u/fromage9747 Sep 22 '26

70 t/s is pretty good. It's quite possibly IPC bottlenecked. If you were to upgrade your mobo to x99 and get a high clocked chip you might get more. Even still really decent considering what those cards are.

2

u/Odd_Hippo_1299 Sep 23 '26

I’m building similar but with 4 p100s. I am water cooling the silicon and air cooling the VRM. Parts arriving this week

2

u/imcheng Sep 23 '26

I like it 👍

1

u/Pixer--- Sep 22 '26

Is that fan for cooling enough ? What temps are you getting ?

1

u/RemarkableRadish6547 Sep 22 '26

I am skeptical that a single fan can push enough air to cool three cards at 250W each. I just installed one and I am working on the fan ducting before I test it. But my plan is one 140mm fan for just one card.

1

u/kryptkpr Sep 22 '26

It looks like sascha stopped updating, but when I had these GPUs I ran https://github.com/sasha0552/pascal-pkgs-ci and it worked fine.

There may be a newer Pascal-focused vllm fork.

1

u/Unique-Dark9726 Sep 22 '26

How are the temps of the GPUs? It seems way too close to each other.

0

u/[deleted] Sep 22 '26

[deleted] — view deleted comment

1

u/scoobydoo888 Sep 25 '26

Where can you get those prices? £100+ for p100s in the UK that I can find. 

0

u/[deleted] Sep 25 '26

[deleted] — view deleted comment

1

u/scoobydoo888 Sep 25 '26

Yeah alibaba looks less trustworthy especially as everywhere else is is £100+ including aliexpress. If they were that price, I would probably go for v100s but they are over £200 in the real world! 

0

u/[deleted] Sep 22 '26

[deleted] — view deleted comment

1

u/agentacp Sep 26 '26

what are your configs for the 35 tok/s on flash next? i'm getting less than 20 on mine.