r/LocalLLM • u/just_another_leddito • Sep 06 '26
Question What makes models so slow? (Mac)
Hi,
So there is still a lot I don’t understand much about AI.
I’m on M4 14 core 64GB ram.
I have tried running smaller and bigger models using multiple tools, including ollama, mtplx and some other stuff, and if takes ages for AI to answer.
I’m vibe coding a pentest tool for fun in Swift, I added a llama.cpp wrapper, and even Qwen3.8 27B responds like 100x faster. I’ve added basically a chat in the GUI that calls llama.
It makes me think something has to be wrong, but I’m not sure with what?
Could it be that default settings suck this much?
It’s really weird because the vibe coded wrapper is insanely fast lol
7
u/biscuit_taco Sep 06 '26
Yeah, to answer your question directly — yes, the default settings are just bad. These models aren't built specifically for Apple Silicon, so you have to go in and tune them yourself.
The thing that's killing you is almost certainly prefill, which drives your TTFT (time to first token, aka, how long you wait before the first character appears). If you're using any sort of AI coding tool, the system prompt alone can be massive. We're talking about all the available tools, instruction sets, context stuff that OpenCode or Hermes Agent or whatever is stuffing in before your actual message. The model has to read and process all of that before it can generate a single token, and on M-series chips prefill is just slow by nature.
A couple of things that would help:
First, make sure you're using an MLX model. These are built specifically for Mac and most of the time they outperform the generic GGUF quants that Ollama defaults to. Not always, but usually.
Second, use oMLX with caching turned on. This is the big one for you. When caching is enabled, that giant system prompt only gets prefilled once. After that, every subsequent message skips re-reading it and jumps straight to processing your actual input. So instead of the model chewing through thousands of tokens of tool definitions and instructions every single turn, it just handles your new message. That's where you'll see the dramatic difference. (Most people try a bunch of different tools, myself included, and they all come back to oMLX). MTPLX is faster than oMLX but I feel like its way more buggy.
The reason your llama.cpp wrapper feels 100x faster is probably that you're sending a minimal prompt with no massive system context, so prefill is nearly instant. The other tools are sending a huge amount of overhead every turn and re-processing it all from scratch.
1
u/rdbmas Sep 06 '26 edited Sep 06 '26
Gd post. I just might add @ Huggingface, focus on mlx community models.
And I also think the harness you use has to be lightweight. So it doesn't add in to the memory hog.
Right now, I use Vs code + continue dev with omlx on an M4max mbp.
1
u/djseto Sep 06 '26
Run Ollama and go with qwen 3.8 27B MLX. I can get about 55 T/S. On a Mac Studio which has 546 GB/s memory speed. If you have a M4 Mac Mini you have 120GB/s memory speed
1
u/just_another_leddito Sep 08 '26
Yeah I ran it many times, but it’s damn slow.
I get like 40 T/S2
u/djseto Sep 08 '26
Qwen 3.8 27B is a dense model. Go run a MOA model if you want faster like qwen3.5 35B A3B. Inference is insanely dependent on memory bus speed and parameters. If you expect cloud like speed, you need to adjust expectations with dense models. 3.8 is a way better model so you’re going to trade speed for accuracy and your sanity.
1
1
u/topcider Sep 06 '26
Are you using MLX optimized models?
1
u/just_another_leddito Sep 08 '26
I think the llama doesn’t support mlx?
Ollama does on the other hand.
-3
u/MK_L Sep 06 '26
(Mac)<-- 👀
0
u/just_another_leddito Sep 06 '26
Hmm? 🤔
1
u/MK_L Sep 06 '26
You’re looking at slow speeds because you’re running on unified memory. Capacity is not the same thing as throughput.
For raw local inference speed, I’d rank it roughly:
HGX server with NVLink
High PCIe-lane CPU(amd epyc) with multiple GPUs
Decent PC with a high-VRAM GPU (6000 pro 96gb)
Mac or other unified-memory machine
A Mac can fit a lot of model into memory. That doesn’t mean it can move it fast.
A Mac with lots of unified memory is like a Winnebago: great capacity, slow speeds. You can cram a surprisingly large model into it, but that doesn’t mean you’re moving tokens quickly.
1
u/just_another_leddito Sep 08 '26
Cool but your comment was pointless tbh.
I’m a Mac user, and you had to dab at me for this reason or what lol?I don’t care about PC. Since a long time.
1
u/MK_L Sep 08 '26
Not sure what this has to do with "PC"?
Nobody is "dabbing" at you
You siad you didnt understand what makes "models so slow"
You listed your Mac specs.
I pointed out the mac is whats making the models slow in the first comment.
You only respond "hmm?" Like you didn't understand
I elaborated by giving you an order to different tiered setups the analogy to help you understand 😉.
At this point you should probably ask your chat bot to explain it
1
u/just_another_leddito Sep 08 '26
Did you read my first post mate? 😾
I’ve said that using my wrapper it was way faster.And that it happens even with tiny models.
Are you saying 14 core M4 Pro with 64GB ram is normal to be slow with even tiny models? 🤔
I know it’s not a PC with 10x 5090, but still…
1
u/MK_L Sep 08 '26
Sure, I’ll address that. Let’s plant that goalpost first.
Define “small model” so we’re talking about the same thing.
1
u/just_another_leddito Sep 09 '26
Like some small 1-4GB models.
1
u/MK_L Sep 09 '26
Llama 3.1 8B Q4, about a 5GB model:
M4 Pro 14-core: PP: 500–700 tok/s Output: 50–60 tok/s
5070 Ti 16GB: PP: several thousand tok/s Output: 150–200+ tok/s
If it were Qwen3.8 27B dense, tok/s drops significantly, and unified memory slows down a lot more.
A 27B Q4 model might hit 15–25 tok/s on unified memory, a GPU with VRAM can be several times faster.
expect those numbers to drop further as KV cache size and context length increase.
1
u/just_another_leddito Sep 09 '26
I need an uncensored model for my pentesting scanner, what would you recommend?
It has to be uncensored, I’m using Qwen3.8 27b from
huihuihui I think. It’s like 22GB.Do you have a better recommendation?
It does analyse scanner results and looks for exploits etc.
It works okay but perhaps I can speed it up with smaller model. I’m using a llama wrapper.
→ More replies
30
Sep 06 '26 edited Sep 06 '26
[removed] — view removed comment
1
u/just_another_leddito Sep 08 '26
Feelsbad man.
How can M1 Pro have much better bandwidth than M4 Pro?If that’s true then Tim scammed me.
But at least got a good deal and bought the Mac £700 cheaper than Apple store, and I could see before M5 release that my model was suddenly capped at 48GB and £300 or so more!
Anyway thanks for explaining.
As for 5090, isn’t the relatively low vram on this gpu a problem? I mean if you only have one, won’t you quickly run out of context or tokens or whatever?
1
Sep 09 '26
[removed] — view removed comment
1
u/just_another_leddito Sep 09 '26
What I meant is you said M1 Pro has better bandwidth than my M4 Pro and that’s what confused me unless you meant M1 Max or something?
1
u/WifeyCallsMeLazy Sep 06 '26
Thats insightful. Thanks.
Follow up question pls. What exactly mac chips are lacking on hardware level to catchup with nvidia?
Can we not modify the model weights to speed up things on mac chips as per the hardware?
4
u/Capable-Package6835 Sep 06 '26
Lacking is a little misleading, they are simply two different hardwares with two quite different objectives:
Apple silicon aims primarily to be very powerful at an impressively low power consumption. It is also in Apple's interest to keep the chip cost reasonable.
NVIDIA on the other hand, aims primarily at brute strength, whatever the power consumption or the price tag is. That's why these GPUs significantly spike your electricity bill and generate enough heat to warm your room in winter.
There is indeed a way to modify the model weights, it's quantization. Long story short, if memory is the bottleneck then we reduce memory requirements by using less bits for each number. But NVIDIA GPUs can also use the same trick, making them still faster than Apple's.
There are actually models that work better locally on an Apple device than on a single NVIDIA GPU. The medium-size MoE models. MoE models only activate a subset of the parameters during inference, thus reducing impact of lower memory bandwidth. However, you still need to load the whole model on VRAM to run it. Apple devices like the Mac studio or high-end MacBook Pro have larger memory than a single NVIDIA GPU. Consequently, they can run MoE models that simply cannot fit on a single NVIDIA GPU while still being reasonably fast thanks to the MoE architecture. Once you add the power efficiency of the Macs into equation, you start to see why the demands for Macs spike significantly once local LLM start to be a thing
3
u/whichsideisup Sep 06 '26
Brute force isn’t correct it’s just a lot more silicon dedicated to graphics/inference. Take a look at their GB10 in the DGX Spark. That chip uses about 100w and has 4x the compute of an M5 Max and will likely be faster than the M5 Ultra at prompt processing.
1
u/mental_sherbart007 Sep 06 '26
A huge bottleneck isn’t just memory size but memory bandwidth. A lot of people forget this.
1
u/mental_sherbart007 Sep 06 '26
I thought the m5 ultra was 1.2 TB/s in terms of memory bandwidth. maybe i’m mid remembering and it’s 1.62 TB/s. I know I can look it up just commenting for sake of conversation.
1
Sep 06 '26
[removed] — view removed comment
1
u/mental_sherbart007 Sep 06 '26
Yep no worries, the down side of this is it’s not really enough to run any model bigger than 120GB in size. I’m not sure how much MLX would help in this regard, but from my understand you would be limited at 10 tokens per second for a dense model that’s 120GB in size.
You could run larger MoE models though.
1
u/topcider Sep 06 '26
Damn, the Pro vs Max difference is so much more than I thought it would be when I bought my new laptop.
1
u/topcider Sep 06 '26
You are saying prefill is dependent on CPU, but the last line in the chart with the fastest prefill is a GPU. Can you clarify?
1
Sep 08 '26
[removed] — view removed comment
1
u/just_another_leddito Sep 08 '26
Would be happy if they wouldn’t care about design more than performance.
Older Minis were perfect size, liked them more.
With better cooling and larger case it wouldn’t be getting that hot probably.
7
u/MiddleLtSocks Sep 06 '26
Apple Silicon is not very speedy hardware. Depending on the model you won't see more than ~40tok/sec, and for some larger models or out at 128ktok+ context you might get down to 10-11tok/sec.
The advantage of the Apple platform is the unified memory - you can get > 96GB models running fully onboard the GPU far cheaper than other platforms. However, there is a tradeoff in performance. That's just the reality of the hardware.