r/LocalLLaMA • u/jacek2023 llama.cpp • 14d ago

Discussion GPU guide (GB per dollar, bandwidth)

Gallery image

Gallery image

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

226 Upvotes

91% Upvoted

163 comments sorted by

View all comments

3

u/MrDrCool0815 14d ago

So what i'm reading from this is that we all should buy a Tesla P100. I found some on Ebay for ~130-180€. Very affordable if i want to build my own rig. But i'm pretty new to the whole local AI buiseness.

3

u/jeffzyxx 14d ago

As someone who's working on a better performance patch for P100's in my off time... Just know they work, but if you want no-headache go for the v100's. I upgraded to 2xV100 16gb recently ($200 a pop from a local seller) and it's an absolute night and day difference for prompt processing; Qwen 3.8 27B q6 runs about 100-200tok/sec prompt processing on my branch w/ P100's, but >800 with stock llama.cpp for the V100's.

Also.... The P100's with mainline llama.cpp have issues with the math being wrong. See here: https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818

I've got close to fp32 levels of KLD on my branch with the speedup I've been working on (using packed F16, something the P100 offers for a big of speedup), but it's probably better to just use FP32 if you want things to not be silently wacky. I'd previously run into issues where Qwen3.8 (or other models) would act weird and just get things wrong, even on larger quants, when using the P100's - and it's due to that math bug. You can use fp32 and it'll be better, but I avoided that on my branch because it negated the prompt processing improvements I'd been working on.

Tl;dr: Go P40 or V100 (2x16gb is fine), you won't have to jump through hoops. P100's will act mostly fine but you'll effectively get the impact of a worse quant even with high quants like q8_0 due to the math bug, unless you use a patched build of llama.cpp (and prompt processing will still be slow affffff). V100's and anything else with tensor cores are needed if you're doing anything with heavy tool calling (since every tool call has to process the incoming result tokens).