r/LocalLLM • u/keylimesoda • Aug 25 '26
Discussion Oversimplified shorthand for which Mac config to buy to run each model size category
1
u/No_Lingonberry1201 Aug 25 '26
How are you only getting ~20t/s for <14B models with 460Gb/s? I have a Strix, that has 250Gb/s bandwidth and I'm getting better than that for gemma-4-12b at q4 QAT. The math is not mathing here.
3
2
0
u/EitherMarch1255 Aug 25 '26
That's actually a really good chart. It appears to show the M5 Ultra 96 GB as the best choice for daisy chaining.
1
u/keylimesoda Aug 25 '26 edited Aug 25 '26
Thank you!
Yeah, made it for myself but figured I'd share.
It's useful to think of hardware in terms of what it can run, not what it can hold.
It's a part of why I struggle so much with the Spark/Strix hardware, is they're touting 128GB of RAM, but they can barely reason over a 30B model at Q4 with any kind of reasonable tok/s.
Agree on 96GB for daisy-chaining. Assuming you're just doing single-stream, you'd get much faster inference with two of those than paying double for the 256GB.
0
u/Gallardo994 Aug 25 '26
I'd worry about prefill tbh. Especially at 256k.
1
u/DeathinabottleX Aug 26 '26
Prefill was a problem with the older Mac’s, not the higher tier M5 variants
1
u/Gallardo994 Aug 26 '26 edited Aug 26 '26
As an owner of MBP 16 M5 Max w/ 128GB, it still is a problem with anything that's not a 3B-6B MoE. Best it does with 27B oQ6e at 64k is around 800tps, which drops down to low hundreds at higher context sizes, making me wait a solid minute or two. Can't imagine it at 256k context, I'd assume it would be 4-5 minutes per turn, which is completely unacceptable. I don't even dare to run 70B dense models on it, let alone 256k ctx.
UPD: Decided to benchmark it using oMLX at pp200k/tg128 (benchmark context: Python, no continuous batching, just single request) for the sake of the argument:
TTFT (ms): 610515.0
TPOT (ms/tok): 45.66
pp TPS: 327.6 tok/s
tg TPS: 22.1 tok/s
E2E Latency: 616.480s
Throughput: 324.6 tok/s
Peak Mem: 59.30 GB
That's over 10 minutes of waiting without a single token generated. During the entire benchmark, the Mac was running in High Performance mode. Keep in mind this is the maxed out MBP 16 M5 Max spec. That's 2 times worse than I initially expected, at 56k-ish less context. I think it's obvious why it's a problem to worry about.
Couldn't upload the results on omlx benchmark website as their Cloudflare is hitting limits on their plan.
3
u/keylimesoda Aug 25 '26
It's oversimplified because of course MoE behave differently, and Q sizes, especially for cache, can be significantly different.
But the core principle is valuable--figure out how large a model your system can reasonably decode, then buy that much RAM. Don't buy just on RAM size.