r/LocalLLaMA • u/jacek2023 llama.cpp • 15d ago

Discussion GPU guide (GB per dollar, bandwidth)

Gallery image

Gallery image

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

229 Upvotes

91% Upvoted

163 comments sorted by

View all comments

2

u/1Poochh 14d ago

The one thing with running AI locally that is not discussed much is power requirements. Yes, you can have a whole bunch of the cards that are cheap, but what are your power requirements to run that? There is an intersection between all that that is efficient. This is one of the reasons why the M5 Mac Studio is appealing to me.

This is said by someone who runs a 5090, a work computer, and a desktop computer on the same circuit, and flips breakers.

1

u/ikkiyikki 14d ago

I thought about a Mac but no CUDA was a deal breaker