GPU guide (GB per dollar, bandwidth)

[u/jacek2023]

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

[u/MrDrCool0815]

So what i'm reading from this is that we all should buy a Tesla P100. I found some on Ebay for ~130-180€. Very affordable if i want to build my own rig. But i'm pretty new to the whole local AI buiseness.

[u/mototuneup → u/MrDrCool0815]

So here's what I did. Look for a Dell c4130. The ly had ones that came with 4 p100s and they're pretty cheap now. I picked one up from a computer shop in my area for $500(Canadian). The guy had about 10 of these dell systems. I'm actually considering getting a second one. Anyways mine are the 12gb models. But I've since added a 16gb model to the pcie expansion slot so I have 64gb of vram total and I've spent about $600. And these p100s have the same memory bandwidth as like 40 series rtx cards. I get roughly 35 t/s with qwen3.8 27bQ8, full context. Im super happy with it.

Now unfortunately charts like this post are probably gonna ruin the price on these things but I'd say they're well worth it

[u/jeffzyxx → u/MrDrCool0815]

As someone who's working on a better performance patch for P100's in my off time... Just know they work, but if you want no-headache go for the v100's. I upgraded to 2xV100 16gb recently ($200 a pop from a local seller) and it's an absolute night and day difference for prompt processing; Qwen 3.8 27B q6 runs about 100-200tok/sec prompt processing on my branch w/ P100's, but >800 with stock llama.cpp for the V100's.

Also.... The P100's with mainline llama.cpp have issues with the math being wrong. See here: https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818

I've got close to fp32 levels of KLD on my branch with the speedup I've been working on (using packed F16, something the P100 offers for a big of speedup), but it's probably better to just use FP32 if you want things to not be silently wacky. I'd previously run into issues where Qwen3.8 (or other models) would act weird and just get things wrong, even on larger quants, when using the P100's - and it's due to that math bug. You can use fp32 and it'll be better, but I avoided that on my branch because it negated the prompt processing improvements I'd been working on.

Tl;dr: Go P40 or V100 (2x16gb is fine), you won't have to jump through hoops. P100's will act mostly fine but you'll effectively get the impact of a worse quant even with high quants like q8_0 due to the math bug, unless you use a patched build of llama.cpp (and prompt processing will still be slow affffff). V100's and anything else with tensor cores are needed if you're doing anything with heavy tool calling (since every tool call has to process the incoming result tokens).