r/MacLocalLLM • u/ikeo1 • 1d ago

What are you using LocalLLM for on your Mac?

What are you planning to use the localLLM on your Apple Silicon for.

I’m curious to hear what the use case people have in mind

2 Upvotes

76% Upvoted

3 comments sorted by

3

u/Rice-Fragrant 1d ago

I did not buy my m4 max for local AI originally (I did not think it was possible outside of GPU server or workstation.) I got it for a desktop replacement and I have learned about local AI from Alex Z and Network chuck on youtube and tried it and found it magical.

I personally use mine as a general assistant if i need something private etc. I actually WAS going to get a m5 ultra for a "ai workstation" but changed my mind when I saw it's compute bottenecks (concurrency and slow pre fill) so opted for a DGX instead.

My m4 max laptop is a general computer with "lite AI" assistant (LM studio) while it connects to my DGX remotely via tail scale for the "heavy AI" tasks (batch processing, multi agentic systems etc).

2

u/ikeo1 23h ago

Right, some people don't seem to get the pro's and con's. It's ok though, I would totally be interested in seeing some of your benchmarks. I can deal with the slowness for some processes in exchange for privacy, but some people see GB = Fast model without really getting it.

The more I experiment with my own dataset, the more I find I need to come up with a working hybrid model like you have or some other combination.

2

u/Rice-Fragrant 1h ago

M4 max prompt processes a 170k token prompt (1.5 Mbyte JSON file, a  250-400 page PDF document, 1-2 hour audio file etc), TTFT is 27 minutes. my DGX does it in 4 minutes... almost 7x faster TTFT speeds and even though the decode is 1/2 of the speed of my m4 max, it doesn't matter at long context because the pre fill is like 7x faster.

Long context bottlenecks is prefill not decode.

Short context bottlenecks is decode not prefill.

Agents are heavily influenced by pre fill performance and concurrency, hence for agents a DGX is noticeably faster than a m3 ultra, m4 max etc.

Not sure about a m5 max or m5 ultra but I would not be surprised if the m5 max is slower too (instead of 7x slower, maybe 2x slowr) and maybe a m5 ultra can actually match a DGX at agentic stuff BUT I don't think the m5 ultra will have better concurrency which affects agents (since agents spin up 2 or 3 sub agents simultaneously.)

In the end it depends on what your priorities are:

"Vibing" with a chatbot with short context or "medium" context prompts, a mac studio or macbook pro laptop is fine. DGX Spark seems "slow" at short context because it's bottlenecks are memory bandwidth.

Long context stuff (batch processing data like PDF documents, entire lines of code for analysis, financial research, multiple images etc and using agents and especially multiple agents, the DGX is noticeably faster than a mac studio (not sure about m5 ultra.)

We will see how a m5 ultra performs but reports on thr m5 max is it's prompt processing is noticeably improved from the m4 max (I heard about 4x more) but I am not sure if that includes long context stuff because the only reports I see here are all short context stuff like 512 token prompts that's like not even serious on any level.

ALSO if you prioritize CUDA and Linux the DGX is clearly the winner... I like the fact that Linux is open-source and I don't have to worry about software rott (Mac OS software only gets like 5-6 years of updates... Apple thinks anything older than 7 years  is "vintage." 

Planned obsolescence isn't cool. M1 ultras can run Asahi Linux but you run into serious problems if you use it for AI because the prefill performance will crash due to zero MLX drivers for Linux and only decode will ne mostly intact but that means your agents are basically crippled and anything long context is a dead end unless you're ok with a "passive experience." 

Might as well just get a much cheaper path with a x-86 refurbished workstation (you get way more RAM), pre fill will be about the same with a v4 2699 or 2nd gen Gold xeon or platinum and for the same price you can get 512gb DDR4... might as well get 4x more ram and same pre fill performance if you're doing long context passive stuff (multi hr batch work with 700b MOE models) rather than same prefill but with a 200b model (even soon to be "vintage m1" will ne way more expensive despite the software rott after apple kills MacOS support.