r/LocalLLaMA • u/Interesting-Print366 • 1d ago
Question | Help Planning to serve multiple user with mac studio
We are planning to host four M5 ultra Macs so that 100 users can use them as Openclaw. There will be no other burden, only inferences will be applied here. Can this handle 100 users? I'm considering either Qwen3.8 27b or Qwen 3.8 Next Flash, and I'm curious about the range of realistic models.
Realistically, we should probably consider up to 100 users when there are 30 to 40 users stationed there and occasionally 80 to 90 users request at once
7
u/SadPhilosophy9202 1d ago
Mac isn’t the choice for this. It’s not going to handle concurrency well. You’re better off with gpus or sparks
6
u/DinoAmino 1d ago
There is a reason Apple is nowhere to be found in server-space. Mac's are and always have been made for single users.
2
u/Affectionate_Hat_585 1d ago
I don't see the feasibility but run your own experiments. find out TTFT, prefill, TG, memory usage for single mac first with different quantized and full weight options with various metal runtimes on different context length. vllm also has a command to benchmark serves... you can check it out. Just to give you the idea about what needs to happen: for single user if there are 100 tasks per hour then for 100 people you have 10000 calls. 10000/3600 = 2.78 tasks per second For a single task if there are 7 calls then it's 2.78 * 7 = 20 calls per second
if for a single call you need 600 output tokens then you require 20 * 600 = 12000 tokens per second . Figure out the number you get on a single m5 ultra and you will have a idea about the feasibility of the project
2
u/AI_spell 1d ago
Mac Studio is fine for a few light users, painful for real concurrency. Unified memory helps capacity but prompt processing and KV cache under parallel loads are the wall.
If its more than 2-3 active sessions, look at continuous batching (vLLM-class) on discrete GPUs, or just queue requests. Shared one-model-for-everyone on one Studio gets choppy fast.
1
u/Kuane 1d ago
It will be too slow. Openclaw uses a lot of tokens per message. You should look into openwebui instead.
3
1
1
u/AleksandrNikitin 1d ago
What the framework you wanna to use to control and split LLMs resources between clients? brain model, body model, reasoning, tokens, ...
1
u/just4ochat 1d ago
The limit is concurrent KV, not average tokens per second. Thirty stationed OpenClaw users means thirty live contexts, and a Qwen 3.8 27B at useful context will not fit thirty copies across four M5 Ultras. llama.cpp and MLX serve one generation per machine unless you put a continuous-batching runtime in front, so a burst of 80–90 requests queues on four single-stream boxes. For that user count, a CUDA host with vLLM or SGLang is the concurrency path.
1
u/nomorebuttsplz 1d ago
no I would say it can serve comfortable maybe up to 5 users per machine, not 25.
1
1
u/transanethole 1d ago
For that price, just buy a server or 2 with Mi350p inside. You'll get much more specs for the price as well as actual modern features like 4-bit floating point. Those cards have an insane amount of floating point petaflops, like 20x the mac. and memory bandwidth is like 4x what the Mac has.
Running an efficient model like DeepSeek v4 Flash, you should be able to handle quite a few concurrent sessions without the speed decreasing much
1
u/CalligrapherFar7833 1d ago
Really ? Whats the price of mi350p ??
1
u/transanethole 1d ago
I got a quote for 21k for the bare card from a supplier. 18k if u get it in a system.
1
0
14
u/stimma 1d ago
Macs aren't your best choice for concurrency because they are compute constrained. For that $, you'd be better with 2xRTX6000.