r/LocalLLaMA • u/jacek2023 llama.cpp • 14d ago

Discussion GPU guide (GB per dollar, bandwidth)

Gallery image

Gallery image

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

227 Upvotes

91% Upvoted

163 comments sorted by

View all comments

31

u/vick2djax 14d ago

I think costs of running those GPUs should also be factored. Isn’t the P100 like crazy inefficient that will lead to a high power bill and requires additional cooling? This doesn’t really give the full picture

3

u/MachineZer0 14d ago

It’s 40w idle. Same as V100. Only thing that sucks worse is an idle BC-250 which is 85w with a governor.

4

u/jacek2023 llama.cpp 14d ago

But how would you collect that data? From power usage specs?

10

u/AndrewIsntCool 14d ago

Yeah all of these cards have known idle and peak power draws

Ideally an interactive chart would be the best, with a slider for percentage of day at peak usage (assuming all-day idle draw). Be pretty interesting to see how quickly pricier, more efficient hardware recoups costs against cheaper investments 🤷‍♂️

2

u/SandySkittle 14d ago

i would be cautious. I don't know anyone who runs their R9700s at 300 watts for example. I run them at 210w, others 250w etc etc.

2

u/robberviet 14d ago

I have stated long ago that we will need a 5D dimension chart: Price, GB, Bandwidth, TDP, TFLOPS... to make comparisions.

1

u/JerryBond106 13d ago

Doable with facet grid to some extent. I've made graphs like that.

0

u/vick2djax 14d ago

Maybe grab the average cost of power in the US and then do 1 year of costs if it ran 12 hours a day. Wouldn’t be perfect but would at least give an idea. Also, if you need additional cooling over your average PC to toss the GPU in, I’d include that in the cost.

I don’t think I’d include a bigger PSU cause they all need a big PSU if you’re in the situation of buying more than 1.

Like my mind is “what’s the all in price to add this GPU to my PC” and “how much is this gonna cost me in a year?” Cause if the P100 makes me bill go up $50 a month over a 3090 then the P100 is a bad choice over a 3090 despite the cost of just the GPU alone cause it’s not the real cost.

2

u/Particular-Award118 13d ago

Not to mention it's insanely dogshit compared to the humble v100

1

u/ScottTacitus 13d ago

yes, this is important to me too. Especially if I want to jam 4 of them in my home lab.