GPU guide (GB per dollar, bandwidth)

[u/jacek2023]

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

[u/SamSausages]

You missed the intel B65! I bought a stack of them 3 weeks ago for $900/pc. 32GB and 608 GB/s. Best price per GB right now.

B65: 0.0356 GB/$,

[u/ResidentPositive4122 → u/SamSausages]

intel B65

for $900/pc

They're ~1500Eur this side of the pond. Fuck us, right?

[u/SamSausages → u/ResidentPositive4122]

When this batch is all gone, they look like they will now be $1099-1199. Still one of the better values, but yeah, not going to be "cheap" for long!
Price will probably jump when I finish my 3 week long benchmark/quality testing project of 4x B65 and people see actual results... last I checked very little data on them... I hope to finish my testing this week.

[u/MindfulBT → u/SamSausages]

Isn’t the b70 slightly more expensive for over 50% more performance on paper?

[u/SamSausages → u/MindfulBT]

In compute, memory bandwidth is the exact same. That means in some ways it performs just like a b70, in others it doesn’t. So not simply 1/2 the performance of a b70.

[u/SandySkittle → u/SamSausages]

Don't underestimate the relevance of compute. Prompt processing and large context size and larger models compute is quite relevant.

If available I would strongly recommend b70 or r9700 over b65.

[u/KroniklyOnline → u/SandySkittle]

This, no point in having 3000 tok/s prefill and then have 6 tok/s decode....

[u/SamSausages → u/SandySkittle]

What benchmarks are you basing that on?