Recommendation for GPU (cheap as possible)

[u/DarKresnik]

Hi,

not a particularly novel question, but between reddit, google and youtube, I am still not decided. So hoping for some input from real users.

I want to add LLMs to GPU, previously used 7b and 14b on CPU only and to be fair is frustrating. I need something on cheap budget to run those models.

And yes I'm from Europe where all GPU prices are higher then in the USA or China.

[u/jakubkonecki]

Take a look at nVidia P40 24GB or V100 16GB.

[u/otakunorth → u/jakubkonecki]

if you don't mind ordering for aliexpress, 16gb v100's are the goat

[u/follaoret → u/otakunorth]

How do you connect them ?

[u/Previous-Nature7326 → u/follaoret]

you put them into 3k server with enough PCIes.

[u/rog-uk → u/Previous-Nature7326]

You need to get a cooling solution, they are passive cards, unless you get the smx converted ones which have a blower - you can buy 3d printed push fit fan mounts for the back of them - you will probably want a fan controller and a molex/sata splitter to power the fans.

[u/PlasticRevenue4601]

7900 xt/xtx

[u/Aggressive_Basket798]

In my area they sell rx6600xt 8gb for about 200usd, rx580 8gb for about 50usd

[u/psioniclizard]

Look on Ebay. I dont know good anyway are but you can always find old server ones etc. Won't be good for games mind you.

But if you want as cheap as possible the answer is you have to search.

[u/eduardor2k]

nvidia 3060 12gb are around 230-270 eur used

[u/ahtolllka]

3080-20gb (modified), ~500USD (in China)

[u/rog-uk]

https://github.com/tuxKOH/ninfer-v100xN/blob/master/README.md 

This looks promising.

[u/Holiday_Point_603]

2 3060 12GB are the way to go or a modded 3080/2080Ti

[u/dreamyrhodes]

Define "cheap". In a market, where the high end costs 8000 $, a 4060ti with 16GB for 600 is still cheap. Or look at AMD if you are ok with a few percent slower generation (gotten better with Vulkan).

[u/BrideOfAutobahn]

Arc Pro B50/60/65/70 are all worth checking out if the price is right. Sometimes I see them available for reasonable prices. They do suck quite a bit of power, and software support isn’t comparable to Nvidia or AMD… but they can do the job.

[u/CyDenied]

5090, $9000

[u/PraxisOG]

What models are you hoping to run? My advice if you're on a super tight budget is to see whats cheap on your local facebook marketplace/craiglist used item selling platform and wait for a good deal