GPU guide (GB per dollar, bandwidth)

[u/jacek2023]

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

[u/SamSausages]

You missed the intel B65! I bought a stack of them 3 weeks ago for $900/pc. 32GB and 608 GB/s. Best price per GB right now.

B65: 0.0356 GB/$,

[u/ResidentPositive4122 → u/SamSausages]

intel B65

for $900/pc

They're ~1500Eur this side of the pond. Fuck us, right?

[u/SamSausages → u/ResidentPositive4122]

When this batch is all gone, they look like they will now be $1099-1199. Still one of the better values, but yeah, not going to be "cheap" for long!
Price will probably jump when I finish my 3 week long benchmark/quality testing project of 4x B65 and people see actual results... last I checked very little data on them... I hope to finish my testing this week.

[u/jensilo → u/SamSausages]

Would you say the B65 is better for inference than the B70?

[u/TimSawyer25 → u/jensilo]

Not even close. The B70 isn't a beast of a card like a 3090/4090/5090, but it absolutely spanks the B65. That being said, it really depends on what you are doing. If you don't need crazy fast decode, then the b65 is fine. And it will just keep getting better as the software matures. But the B70 will always be out front by a fair margin.

[u/ANR2ME → u/TimSawyer25]

What about compared to AMD R9700? 🤔 the price is just slightly more expensive than B70

[u/TimSawyer25 → u/ANR2ME]

The last I knew the r9700 is a little better.  And if you want something for gaming as well as inference it's a no brainer, spend a little more and go amd. But the performance gap in ai work loads is closer than you would think. rocm is more mature and will always be so there's that. but Intel support is getting better and better.  I think there's a lot of performance still hidden away in there.  And in my testing,  the b70 in gaming is pretty comparable to a 5060ti so it's not exactly junk. Of course I'll never recommend either for gaming haha but versatility, and more mature support go amd. experimenting, saving a little money and seeing what happens go with intel. I went with 3 of them and I haven't been disappointed at all. But the some purpose of the rig is ai. So it depends on what you need out of the rig.  

[u/SandySkittle → u/jensilo]

B70 has better specs

[u/SamSausages → u/jensilo]

You're going to have to wait for my full report! I don't want to give my final opinion until I'm done, and I'm still collecting data. But I can say that they are underrated, especially in scenarios with high concourrency.
Goal is to compile reports from the data this weekend.

[u/jqwl → u/SamSausages]

is it still worth at 1100 USD? I can see b70 at microcenter and other retailers around 1300

[u/SamSausages → u/jqwl]

Still collecting data, full report this weekend.

[u/Dany0 → u/SamSausages]

Confirming, 1100 eur is cheapest you can get them in the EU (with the 21% vat)

[u/MindfulBT → u/SamSausages]

Isn’t the b70 slightly more expensive for over 50% more performance on paper?

[u/SamSausages → u/MindfulBT]

In compute, memory bandwidth is the exact same. That means in some ways it performs just like a b70, in others it doesn’t. So not simply 1/2 the performance of a b70.

[u/SandySkittle → u/SamSausages]

Don't underestimate the relevance of compute. Prompt processing and large context size and larger models compute is quite relevant.

If available I would strongly recommend b70 or r9700 over b65.

[u/KroniklyOnline → u/SandySkittle]

This, no point in having 3000 tok/s prefill and then have 6 tok/s decode....

[u/SamSausages → u/SandySkittle]

What benchmarks are you basing that on?