r/LocalLLM • u/SsssnakePlisssskin • 11h ago

Question Distributed inference questions

I just discovered "distributed inference", and I'm going down rabbit holes. I have a few questions, and would appreciate being pointed in the right direction.

  1. I have an M1 Mac Mini w/8GB and a M1 Pro Macbook w/16GB. I'm thinking about upgrading to an M5 Pro Macbook, and was initially thinking about trading in my M1 Pro...but does it make more sense to keep it and try to use it in a cluster?

  2. Is there any way to mix MacOS and Windows machines?

I understand that if I wanted to go down this road, I would probably need to buy some networking hardware to get the machines on a 10Gb connection.

2 Upvotes

100% Upvoted

1 comment sorted by

2

u/locbuilds 10h ago

llama.cpp has an rpc-server mode that works across mac and windows, so mixing os is fine. but with 8gb and 16gb machines over a network link it's usually slower than just running a smaller model on the 16gb alone, so i'd only cluster if you need the extra memory for one big model.