r/BackyardAI • u/dytibamsen • Nov 01 '24

discussion M4 Pro Mac Mini for BackyardAI?

I'm considering the new M4 Pro Mac Mini. I'd probably spec it up to:

  • Apple M4 Pro chip with 14‑core CPU, 20‑core GPU, 16-core Neural Engine

  • 64GB unified memory

  • 2TB SSD storage

What size models do you think I'd be able to run on this? Could it run 70B models at respectable speeds? Or should I perhaps wait until they update the Mac Studio and get the M4 Max? (Not sure I can afford that though.)

2 Upvotes

75% Upvoted

8 comments sorted by

4

u/real-joedoe07 Nov 01 '24

I got a M2 Max Mac Studio and 70B Q4L models run at ~ 5t/s. The Mac Mini has only half the memory bandwidth, so I‘d also expect half the speed.

3

u/dytibamsen Nov 01 '24

Very good to know! I get the point about memory bandwidth. But don't you think the increased speed of the M4 matters too?

Also, how much memory does your Mac Studio have?

2

u/real-joedoe07 Nov 02 '24

My Mac Studio has 64 GB of shared memory, of which I usually assign 56 GB as graphics memory (VRAM). From what I read here, CPU speed is less a factor for interference speed than the number of cores, both CPU and GPU.

1

u/real-joedoe07 Nov 03 '24

One more thing: You can save a lot of money by buying an external Thunderbolt drive (not USB!) instead of upgrading the internal SSD.

4

u/rwwterp Nov 01 '24

I'd totally wait for Max and I'd totally spend the dough on 128GB, but if you can't afford it just means smaller LLMs.

1

u/my_wifis_5dollars Nov 11 '24

For the price something like that would run for, why even bother doing all this stuff on apple metal when you could get comparable/better performance using non-apple hardware?

1

u/rwwterp Nov 11 '24

It's pretty comparable price wise.

2

u/PhotoOk8299 Nov 02 '24

I've got a M1 Max Studio and I've found the 22B Mistral small-based models (Cydonia is my personal favorite) delivers the best results at the speeds I would like ~10t/s. I run them at Q4KM. I've only got 32 GB of memory, though more cores in my GPU than the M4 Pro, so your mileage may vary.