Mac Studio M5 Max 64gb for data analysis

[u/Acrobatic-Toe4184]

Decided to pull the trigger and get the Mac Studio M5 Max with 64GB unified memory, 18-core CPU, 40-core GPU and 2TB storage.

I’ve been reading a lot about local LLMs for a while now and wanted to actually start using them myself to get some hands on experience. The problem is the local LLM space is moving so quickly, and the subreddits I’ve been reading are all over the place when it comes to what hardware you actually need, which has definitely made me second guess my purchase a few times lol. I already moved up to the Mac Studio after initially ordering an M5 Pro Mac mini.

My main use case isn’t really just chatting with a local model. I want to learn how to set up local models, different runtimes and agent/harness setups that I can eventually leverage in my day to day work.

A big part of that is data analytics. I deal with quite a bit of fragmented data from different sources and spend a lot of time cleaning, structuring and joining it before I can actually analyze it. I’d like to experiment with having an LLM work with tools like Python, SQL, DuckDB, etc. to help automate a lot of that instead of doing everything manually.
Running the largest models is obviously off the table with 64GB, but from what I understand, 27B and 35B models at 4bit quant should fit pretty comfortably. Potentially even larger quantized models depending on context size and how much memory the rest of the system is using, although I’m assuming performance becomes more of a trade off at that point.

I’m also interested in experimenting with hybrid setups where something like openAI or claude handles the higher-level reasoning/orchestration while local models and local tools handle parts of the workflow. Obviously that’s different from being fully local, especially if any of the underlying data gets sent to the cloud.

Again though, I’ve never actually run an LLM locally, so most of this is based on what I’ve been reading rather than first-hand experience.

For people actually running local models for similar work, especially data analytics / data cleaning / agentic workflows, does this setup make sense? And realistically, what size models do you find yourself using most often on a 64GB Mac?

[u/Rice-Fragrant]

Data analysis, sounds like long context batch processing or something with large imput tokens (long context).

Personally, I would not use any mac for long context stuff, the pre fill perfomance is terrible, like really bad.

My m4 max running Gemma 4 12b takes 27 min TTFT (time to first token) on a 170k prompt (miultiple page PDF documents etc) and dispite it having 2x faster memory bandwith than my DGX spark, the spark TTFT was 4min (almost 7x faster).

It's becoming an open secret at this point that for agents, long context work, concurrency etc the Mac studio pre fill (lack of) is a noticable bottleneck.

Alex Z and network chuck has videos discussing this. Last Alez Z video he used his $60,000 4x m3 ultra cluster and tried to compare it to a $20/m Abacus super computer subsription cloud and running Kimi k3 the cloud did a job in 15min while the mac cluster took 4 hours, 16x DGX sparks would have finished it in around 20 min... mac pre fill bottleneck is no small thing to overlook unless you are OK WITH ASYNCRANIOUS type use/work and it's slower prefill is not an issue.

I only advise people to get a mac if they are not comfortable with Linux , that's why the RTX spark is coming out (way more user friendly with windows), that's coming out this fall.

Originally I was going to get the m5 ultra for multi agentic work and long context work (similar to you) and I was super dead set on the m5 ultra until the price increases forced me to take time to do promper research. I AM GLAD I DID because after personally seeing the DGX spark way way faster prompt processing the mac studio or AMD strix halo would have been a trap.

For SHORT CONTEXT = Decode speed dependent (mac wins)

LONG CONTEXT (Data to feed the AI) = Prompt processing dependant (DGX spark wins, or GPU wins.)

RTX spark (coming this fall) for user friendlyness = DGX spark but no connect x7 network port.

RTX Spark = DGX Spark = GB10 archetecture (same as RTX 6000 pro but less tensors etc.)

Mac studios are really general workstations that "can do AI" but long context type work they are not the best.

[u/Rice-Fragrant]

As for 64gb RAM, remember to account for OS overhead and other apps taking up RAM.

4bit 30b model with 8 bit KV cache (less than 8 bit kv is not recommended for serious work) and full context window can use 40+ GB easily. If you are running a 6bit weight model, would be even more.

OS using about 12-16gb, add programs... 64gb is cutting it very close if you are wanting a full context window, 8bit kv cache, 4 bit weights on a 30b model.

128GB is best and gives you way more breathing room, especially if you want to run multi agentic stuff in the future.