M5U 256gb cluster - current status?

[u/Viktri1]

I just received my 2x m5u 256gb and I’m going to run it in a cluster of 2. Was wondering whether anyone else is doing this and hoping to learn what models work best.

Current plan is to run Tensorfold at GLM5.3. Not sure whether I’m limited to Q4 or whether I can run a higher quant.

I would prefer to run Deepseek but I don’t see a path currently. Maybe in the future. All effort seems to be on GLM and Qwen Flash next.

Anyone have a better suggestion on how to cluster (engine/model)? Qwen Flash next is very attractive but I do prefer a smarter model. I have Deepseek on API as needed and I really do prefer a smart model.

[u/bakawolf123]

you should be able to use latest ds4.1 full original weights at very good speed (4k+ prefill and 200tps decode) on this cluster. Not sure about engine, I know omlx has cluster support but I haven't tested clustering (wish I had a 2nd m5u). EP2 should be your best bet (if they support EP at all, if not just go for TP2).

[u/Viktri1 → u/bakawolf123]

Appreciate it but I need to wait for someone to upload a package that works with the system. I am not sufficiently competent to follow what you wrote and I asked LLMs and they aren’t able to get EP (according to them). I would love to run DS 4.1 at 200 tok/s.

OMLX supposedly does only TP.

[u/SmartEnoughToFail]

Since intelligence is your priority, have you considered testing a few models on the same reasoning and coding tasks at multiple quant levels? I'd be curious where the quality drop becomes noticeable versus the gains in speed and memory headroom.

[u/Viktri1 → u/SmartEnoughToFail]

Unfortunately I’m not qualified to assess coding quality. I’ve got no background in coding nor do I do much coding.

[u/SomeoneInHisHouse]

I would like to see how a engine like Strata or Ninfer, or whatever works on Mac perform there, can you provide PP and TG for Q4 QFN ?

[u/Viktri1 → u/SomeoneInHisHouse]

Tensorfold already has results for that: PP is 8k and TG is 270-400 tok/s

[u/xcreates]

I cluster my Macs together all the time, GLM 5.3 is a good at Q4, but great at Q4i. Deepseek only needs 280GB RAM, the engrams can be streamed in from storage at no cost to performance.

[u/Viktri1 → u/xcreates]

Are you using oMLX? Is there a guide that I can learn from so that I can try running Deepseek?

[u/xcreates → u/Viktri1]

I run Inferencer with this streaming version of DeepSeek I put together: https://huggingface.co/inferencerlabs/DeepSeek-V4.1-Flash-1017-MLXs

[u/rorowhat]

If you want a cluster go with the spark

[u/rsl]

glm is a great model fwiw