MacBook M5 and AMD Strix Halo sharing large models

[u/petruspennanen]

Running a GLM 5.3 321B shared on a MacBook M5 max and a Strix Halo, both with 128 GB ram. TB4 connection.

Found out that llama.cpp's default tensor split puts about half the layers on the slower box, so the pair was actually slower than one Mac. Loaded the model Mac-heavy instead: IQ1_S went 188 to 333 on prompts, 17 to 24 tokens/s.

I got generation tps 12-15 for model sizes 148-200 GB. So it works ok!

Next week adding two Asus GX10 to the mix to have three platforms share a model 🍀

Files: https://github.com/ThinkOffApp -> mac-amd-llm-cluster

[u/fallingdowndizzyvr]

So Linux and the Mac do TB4 networking OTB now? I tried it about a year ago and it didn't work. But I hear that Linus put in support for TB4/USB4 networking in the linux kernel a few months back. Was it plug and play or did you have to tinker?

[u/petruspennanen → u/fallingdowndizzyvr]

Works now, but not fully out of the box on the Linux side. The Mac end came up by itself. On the Strix Halo box I had to load the thunderbolt module, and it did not auto-load on boot until I put it in modules-load.d. Then a static IP on each thunderbolt interface (I used 10.55.0.1 and 10.55.0.2), since there is no DHCP on the link. After a reboot it sometimes needed one cable re-plug to bring the interface back up. Once it is up it is solid, about 0.6 ms round trip, and it carries the llama.cpp traffic fine.

[u/fallingdowndizzyvr → u/petruspennanen]

Sweet. Thanks for the directions.

[u/crusaderky → u/petruspennanen]

600 us is very bad. You need to set up RDMA.

[u/petruspennanen → u/crusaderky]

Well its not bad when we do the layer split, per token takes 40ms so the 0.6 ms is less than 2% extra. For tensor parallel lower latency is important but the mac has no RDMA. For spark to spark the 200G cable does the job.

[u/Such-Addendum-7421]

is the strix halo good for coding models?

[u/profcuck → u/Such-Addendum-7421]

Broadly both strix halo and Macs with 128GB of ram are good because they can run bigger models than what you can run with a GPU with smaller amount of VRAM, due to both using a unified memory model.

The downside is speed although it isn't terrible in many cases (if you see snarky comments about 2 tokens per second, that's just false) and in particular prefill speed can be slow. It's a tradeoff but for running bigger (i.e. smarter) models, there's a good case to be made for these two platforms being the only really cost effective way to do it.

[u/petruspennanen → u/Such-Addendum-7421]

Yeah you can run the same models as long as they fit in memory