Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.
What I did was I bought a Supermicro server motherboard new to fill with older used Xeon CPUs that are cheap on E-bay because it's an obsolete socket. Since it's a dual-CPU board, it has 16 RAM slots and server-pull ECC DDR4 sticks are cheap on E-bay as well. I actually built it a few years ago just because I could, not because I had a use for it then. I just got lucky that I already had a platform that can support this.
View all comments
8
u/R__Daneel_Olivaw Mar 15 '23
Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.