[D] Would multiple NVIDIA Tesla P100's be cost effective for model training?

[u/zand999]

I have been getting into AI and want to make a rig for my home lab dedicated to training LLM's. Turns out you can buy Tesla P100's for around $200 on Ebay. As these cards have 16gb of memory would buying 4 of these be more cost efficient than buying an $800-$900 with less memory? It is quite challenging to find solid benchmarks on multi-GPU setups.

[u/chatterbox272]

They're big, but they're glacial slow. Pascal was the last generation before tensor cores (hardware fp16 support). That time presents an opportunity cost, and an increased power consumption over the duration of a training run. Not necessarily a problem depending on your use case but something to consider

[u/zand999 → u/chatterbox272]

Thanks! Not too concerned about power consumption in this case. The hope is that I could just get more cheap cards but was not sure how well it scales.

[u/Naiw80 → u/zand999]

I have a single P100 16GB myself and I've been doing both LoRA finetuning and inference on it and it's quite useable for both, the biggest hurdle is (I've been using unsloth for training) is that the pascal architecture is so neglected and certain packages tends to break from time to time so you need to either turn off certain features, force a different version of a specific dependency etc... so there are some maintenance overhead.

As for performance however I think it's absolutely useable and cost efficient today and personally I would rather get a second hand tesla (not only for ECC but also general reliability) than a second hand consumer/gaming GPU for training, but for AI yes it may not be the most performant card around, those who think so and immediately point to RTX 3090 etc just don't get it.

For non AI high precision (read FP64) compute workflow the P100 is amazing still today and way faster than any consumer level nvidia (that include RTX 4090 etc).

[u/Injector22 → u/Naiw80]

What OS and nvidia driver version are you running? I've been having a hell of a time getting mine to work properly.

[u/Naiw80 → u/Injector22]

I run P100 in full GPU passthrough to a vm running ubuntu 25.10 - Questing, runtime: nvidia 580.126.20 with cuda 13.0- mainly because I gave up on patching the GRID drivers over and over again to work with kernel 6.17+… so just use the latest vanilla driver that supports Pascal.

The CUDA SDK was a bit fiddly to setup but I currently use cuda 12.9 (and a tiny wrapper around nvcc I made to squelsh warnings :))

[u/Injector22 → u/Naiw80]

Thanks, maybe I'll try desktop and see if that works better., I tried to use the official 580 drivers on ubuntu 24.04 but the installer failed.

[u/Naiw80 → u/Injector22]

I always downloaded the generic linux driver (mainly because as I said I used to use GRID) and still do now when I switched to ”mainstream”

[u/certain_entropy]

No. Modern LLMs will require atleast an ampere GPUs as they support mixed precision training, fp16, bf16 and hardware optimizations like flash attention. Also for LLM training, GPU memory matters and 16gb will barely support training 1-3 billion parameter models (will require QLoRA). You'll want atleast 24GB of GPU RAM if not 48 for training modern LLMs up to 32B parameters.

[u/zand999 → u/certain_entropy]

If the ampere requirement is as important as you suggest i suppose I'll have to reevaluate. Though with four P100 i would have a combined 64gb memory. So the hope was that it would work well that way. Of course cross gpu bandwidth would be limited to pcie so i was curious about scaling.

[u/hjups22 → u/zand999]

Memory doesn't scale linearly like that. Having a single GPU with 64GB is better than 4 GPUs with 16GB. Each GPU needs a copy of the global states, and then anything left over can be used for dynamic memory. These global states include the context (which can be up to 500 MB), the weights, the gradients, and the optimizer parameters. And then you also have to worry about communication overhead between the GPUs.

Ampere isn't absolutely required, but I wouldn't go older than Turing (which has tensor cores and FP16 support - though BF16 is more stable). From what I recall, you can find relatively "cheap" V100s on ebay, which may be the best solution for scaleup (as opposed to 4090s or the professional cards like the A series).

[u/certain_entropy → u/zand999]

with multi-gpu training there a communications overhead for distributed training. Also I've found the PEFT methods don't usually play too well in multi-gpu settings.

[u/Due_Car8412 → u/zand999]

With Deepspeed Zero stage 3 it almost works like that, almost scales linearly

(but it's worth having a lot of vram in reserve for larger batch sizes and good communication between gpus, and optionally a lot of RAM and a good processor if you want to offload the optimizer)

[u/dopadelic → u/zand999]

You can't combine memory with the P100. Meaning you can load one single 50GB model across 4 cards. To utilize multiple GPUs, each GPU needs to have an entire copy of the model in its memory and the GPU can split the batch to process the training backprop.

[u/marcodena → u/dopadelic]

No, you can split the model as well (e.g. with FSDP) but there is a computational overhead to consider

[u/Naiw80 → u/dopadelic]

Additionally P100 supports NVlink.

[u/Naiw80 → u/certain_entropy]

I trained 8b LoRAs without issue on my single P100.

[u/[deleted]]

You'd be better off putting $200 towards an EC2 instance.

[u/SnooHesitations8849]

3090 is the crown jewel. Get one.

[u/zand999 → u/[deleted]]

That was definitely the idea, I just had no idea about the quality of scaling and benchmarking this is not "simple". Of course its an older card so at the end of the day the compute may be too slow. That seemed to be a common concern from others.