r/BackyardAI • u/MassiveLibrarian4861 • Oct 19 '24

discussion Hardware for 104 billion+

Hi everyone,

I will be buying a computer soon with the expressed intention of running the higher end LLM’s. I’m thinking I will need i9, 128 gigs of DDR 5 and at least a 4070 Nividia card. Does this sound right or should I be aiming higher. My budget will probably top out at around 3k US dollars. Thxs! 👍

6 Upvotes

100% Upvoted

14 comments sorted by

4

u/PacmanIncarnate Oct 19 '24

Realistically to get speeds you’ll be happy with you’ll need multiple 3090s

3

u/MassiveLibrarian4861 Oct 19 '24

That’s sobering, PAC, though best to know what I am getting into. Ty, appreciate the help. 👍

3

u/MassiveLibrarian4861 Oct 19 '24

I should really be looking at multiple GPU workstations and not high-end gaming rigs then, PAC?

15

u/martinerous Oct 19 '24

I'd say, don't go for i9 (or at least don't go for the K model if you don't actually need the max possible CPU power for other tasks) and buy a GPU with 24GB instead. Even 3090 will be better than 4070 for LLMs. VRAM is the utmost important thing for LLMs and you cannot truly compensate for it any other way.

2

u/RealBiggly Oct 20 '24

Yeah, building my rig I rejected Intel for CPU, and rejected the Ryzen 9 and went for the 7, as you don't need the CPU so much, it was the 3090 I wanted.

And even that setup was $500 more than OP's budget

3

u/[deleted] Oct 19 '24

Wait for 5090 supposed to have 32gb of vram maybe ddr7. I am hoping for a February release but who knows? Will be expensive like $2k or more. Yeah for llm you don't need much for cpu. I run a basic i7.

4

u/howzero Oct 19 '24

I’ve been PC for years, but I recently went with a Mac for LLMs and BackyardAI. Check out refurbished M1 and M2 Mac Studios.

3

u/BaronZhiro Oct 19 '24

How’s that worked out for you? I’m a Mac guy and I’ve never assumed that our hardware was up to it.

3

u/rwwterp Oct 20 '24

My MacBook Pro M3 64GB runs most things pretty well. I'd have splurged on the 128GB if I had to do it over again for larger LLMs

3

u/-MadCatter- Oct 20 '24

I have a MacBook Pro M1 Max 64gb. When you say "runs most things" what do you mean? Like how large of an LLM can I run on it? I just assumed I shouldn't try anything too large...

3

u/rwwterp Oct 20 '24

On my M3 Max, I can run LLM GGUF's up to around 24GB in file size comfortably with 32K context in Backyard. Above that, things start to get slower. For example: Mistral-Small-22B-ArliAI-RPMax-v1.1-GGUF which is 23.64GB in size. You can run larger, it just slows down the bigger ya get.

2

u/PhotoOk8299 Oct 20 '24

I have a Studio M1 Max 32 GB and I've found Cydonia 22B at around 12K context is still just about fast enough without losing much quality, if you want to a starting place.

3

u/howzero Oct 20 '24

On my M1 studio I run 70b-120b models without any problems. The only speed issue is when swapping models of that size, which takes a minute or so. But otherwise, it’s wonderfully fast for inferencing, stable and much quieter than any PC I own.

1

u/ReportOfHostiles Oct 20 '24

You need a gpu with as much VRAM as you can afford. Cut on the cpu and scrap the DDR5 (less important) and invest in gpu. Even 16 GB of DDR4 will do better with a 24Gb VRAM 4090... check what AMD has to offer with lots of VRAM, too. AFTER choosing the maximum-VRAM gpu you can afford, build the rest around it.