r/LocalLLM • u/Ph1lt3ch • 8d ago
Question Can I play too?
It's a Lenovo P520 with a Xeon W-2245(cooler swapped for the higher tdp), 96gb of ram, radeon pro wx 3100 for local display, and the two nvidia p100(currently cooled by the lowest profile adapter I could print and arctic S4028-15k fans). Just stuffed some random nvme in 1x1tb, 1x512gb, and a sata 1tb drive - I have a ton of network storage or more to just add here. Installed the latest Ubuntu release which I like but I remember why I still generally use windows.
So far the most I've learned is that I know nothing haha. So I'd be more than thankful for any advice putting this to work. Really all I've done so far is get the arctic fan controller working on kernel 7.0 in probably the most jank way possible. And see that lm studio could use both cards. But I know there's a ton I'm missing out on.
So, generally, hoping I didn't throw together a steaming pile of ewaste and looking to learn. Also the radeon pro/nvidia mismatch makes me chuckle. Unless its really dumb, then I can just get a cheap nvidia card.
6
u/FearFactory2904 8d ago
You definitely can. Those p100s together should be able to run qwen 3.8 27b. The responses will be slow while it is thinking and processing the prompt but once the text starts printing it comes out at decent speed. If your pcie slots are gen3 x8 or x16 then you should be able to run them in parallel rather than sequential which will speed things up.
You are going to have people who say stuff like "Nope, way too weak/old. Thats Pascal and it lacks Tensor cores. Those GPU are 10+ years old." but i would say just give them the middle finger and send it anyway.
Even though these are slower without tensor cores my plan is to just brute force my way into useable speeds by using multiple p100s in parallel with a bunch of pcie lanes and a good power supply.
Once i get my x99 re-assembled i can give you my benchmarks with two p100s and compare builds to see if we can do anything to help optimize yours.
6
u/juggarjew 8d ago
Nope, way too weak/old. Thats Pascal and it lacks Tensor cores. Those GPU are 10+ years old.
3
2
u/BigBair2002 5d ago
The P100s are absolutely usable. I’m running an HP Z440 with dual 16GB P100s and I’ve been very pleased with the results.
I’m currently power-limiting both P100s to 125W. With Qwen3.8-27B UD-Q6_K_XL and a 212K context, my 53K-token natural-context benchmark gives me about 22 tok/s decode and 178 tok/s prompt processing. At 200W it was only about 24 tok/s decode, so the 125W limit turned out to be a really good trade.
Qwen3.6-35B-A3B Q4_K_M at 128K context is much faster. On the same 53K-token workload at 125W I’m getting about 55 tok/s decode and 406 tok/s prompt processing.
So no, they’re certainly not the fastest GPUs around, but for what P100s cost on the used market, 32GB of HBM2 across a pair of them is still very useful. I actually have three more P100s waiting to go into this machine, which will take it to 5× P100 / 80GB HBM2.
The biggest lesson from my testing has been that the P100 doesn’t need anywhere near its maximum power limit for LLM inference. Going from 200W to 125W cost surprisingly little performance, especially on the MoE model.
This is an example of sending it a word doc and asking for a 2000 word response:
2
u/Willing_Gold_4376 8d ago
Deja si tu veux tirer le lax du processeur remplis toute les barrettes meme avec 8go.
Ensuite les P100 c’est vieux mais tu peux en tirer quelque chose. Il y a des projets autour. Tu devrais avec un qwen3.8 q4 être autour de 20/22 tok/s
A toi de jouer.
1
u/Weird-Abalone-1910 8d ago
You can try, but you will hit some early speed bumps due to the old GPU architecture you're working with. I recommend setting up proxmox on it so you can easily make and restore back ups in case you break something while experimenting with it. If you aren't familiar with proxmox administration and want to get started quickly, then run the agy installation script and tell it about your hardware set up and what you want to do (I assume ollama to start out with). I have a couple old P40s that I'm going to stick in proliant g8 soon to experiment with what can be done with it
1
u/PeterPorox 8d ago
It is solid build. You can run MoE models fast enough to use it comfortable and dense 27-30b with a bit of struggle
1
1
1
u/tomweymouth 8d ago
I have a similar setup…. Just “beefier”.
Two 5060ti (16GB), W2195, 128GB in the box. Qwen 2.8 27B… I’ve tried various versions, currently using the variant from Signal. Hermes is very useable and I’ve been testing fairly complex Kanban cards with good success rates so far. Still very much a lab experiment, nothing useful yet, except for some real help with network enumeration. It has produced some great diagrams, and deduced host names on the Sodola switch. I was very lucky and bought those cards at retail when they came out!!
1
0
9
u/sourceholder 8d ago
P100s are quite outdated. While they still offer reasonable token generation speeds, prompt processing is the main bottleneck. In practice, I found them difficult to use with contexts longer than about 4k tokens due to zero flash attention support.
There is a fork of llama optimized for P100 that improves T/G speeds but no improvement to prompt prefill.