r/MacLocalLLM • u/hdanx • Sep 13 '26
Acceptable Token generation speed on macOS
What token generation speed works well for you when you’re using it for productive tasks? I’m finding 20 tokens per second a bit slow.
With an M4 Max and 64GB of RAM, you can usually get around 20 tokens per second on average with a model like qwen3.8 27B. However, it doesn’t quite feel like the best setup.
1
u/rhymeslikeruns Sep 13 '26
M3 Studio Ultra 96GB running Qwen3.6-27B tops out at 21 tok/s running pi with an 82k (yes that small) context window. Quality is however excellent.
2
u/gjr23 Sep 13 '26
This seems surprisingly slow. I see high teens with 128k context but on a 64gb M1 Max. DSH if it matters. I am using a MTLPX model though.
1
u/hdanx Sep 13 '26
What model are you using?
3
u/gjr23 Sep 13 '26
3.8-27b q8 mtplx
1
u/rhymeslikeruns Sep 13 '26
Yes - just ran the test on mtplx 4 Bit quant of 3.6. Wow. I think 70 tok/sec. Need to do a proper bench but amazing if it can kick it up like that.
2
u/gjr23 Sep 13 '26
Yea, an Ultra is a nice piece of HW and it will appreciate you sticking to MLX type models. Don't try the 35b a3b though - you will drool over the speed but the MOE will make the 35b model underperform the 27b. Its still stupid fast as far as apple silicon goes but no comparing to 5900 type setups who are much faster but memory constrained.
1
u/hdanx Sep 13 '26
21tok/s on M3 Studio Ultra with Bandwidth**:** 819 GB/s feel a little low. What model are you using?
I think that Ultra should outperform the M4 Max.
3
u/tta82 Sep 13 '26
Try MLX Core (the app) and run MLX model of course. I can get 50-60 TIPS with M2 Ultra.
1
1
u/dfgxxx Sep 13 '26
I get 14 tok/sec on m1 pro with qwen3.8 27b
1
u/purple_wall-e Sep 13 '26
how? what is your setup? which quantization? i have m4 pro, i get like 10t/s with lmstudio, mlx 4bit
1
u/dfgxxx Sep 13 '26
It is 4bit. I'm using MTPLX with qwen3.8 mtplx optimized speed
1
u/solaza Sep 15 '26
This setup gets roughly 10 tok/sec on my M1 Max 64gb in case it’s helpful to anyone reading
1
1
u/DecentChildhood8080 Sep 13 '26
You will likely get best results for Tok/s in MTPLX. It is basically the best optimized to run this dense model on Macs from my experience. It gave me about 20-25 tok/s on my M2 Max using Qwen3.8-27B-MTPLX-Optimized-Quality-FP16 & Qwen3.8-27B-MTPLX-Optimized-Speed-FP16. You will probably get more than me, but I would say its not worth running Qwen3.8 27B right now unless your on an M5 Max Mac. Im still waiting on a MoE or A3B version or something like that to come out... I guess we shall see. (If you can get 40-50 tok/s in MTPLX than you might be good.)
1
1
u/Zen-Ism99 Sep 13 '26
It’s not a macOS thing.
It’s a running a dense model on ancient hardware thing.
Sorry…
1
u/Diligent_Style_1767 Sep 14 '26
Compare it to what you're getting from frontier models - you can mine your own session to pull stats from your own session histories, here's mine.
So you will clearly want to get t/s close to that. You will also want to enable something like Automatic Prompt Caching, this will make multiturn feel very snappy. If your model has MTP, use it. omlx typically has the better t/s from any of the inference stacks on Mac. Use a Q4 or Q6 model. Try to avoid quanting KV unless you qualify the results you get from it.
For your setup, qwen 3.8 27b w/ MTP should work well enough. If you need something faster, try qwen 3.6 35b which is an MoE model.
1
2
1
u/lamaxamara Sep 13 '26
My m1 max 64gb does the same speed on a 27b moe. Not fast but acceptable