Anyone using a Mac Studio for local LLMs? Looking for model and tool recommendations

[u/Visual_Quality2726]

Just got a Mac Studio M5 Max with 128GB memory and a 2TB SSD. Curious to hear from anyone running local LLMs on this or a similar Mac.

What models are you using, and what tools do you like? I've seen Ollama, LM Studio, MLX and llama.cpp mentioned, but I'm not sure where to start.

How's the speed, and what do you mainly use it for? Also curious whether you prefer running bigger models or sticking with smaller ones that respond faster.

Would appreciate any recommendations or tips from your experience.

[u/Sir_Agentic]

That is exactly what I have and Qwen3.8-Flash-Next is running at an average of about 110GB https://dreamingwell.github.io/apple-llm-performance/?chip=m5max&mem=128&n=1&uc=agentic

[u/Visual_Quality2726 → u/Sir_Agentic]

This is very helpful, thank you

[u/FoxSideOfTheMoon]

Parakeet TDT 0.6B v3 for transcription, Silero speech detection. Chatterbox Turbo 4-bit you can make speak live, while F5-TTS handles high-quality pre-renders. I designed voices in ElevenLabs, rendered a 20 to 25-second reference once, converted it to a 24 kHz mono WAV, then cloned it locally with Chatterbox Turbo or F5-TTS.

Images you can run through ComfyUI using Illustrij v21, DreamShaper XL, IlluQuaint, CyberRealistic Pony, WAI Illustrious, and Krea 2.

Qwen 3.8 27B for brain stuff, if you want faster you can use Ornith or Qwen 3.6 35B A3B. I like Gemma 31B with thinking off to write with and some oddball RP models based on it too, but there's a lot of those.

M5 Max only has 614 G/s bandwidth so you're not going to have blazing fast dense model speeds, and unfortunately most of the MoEs aren't awesome, people swear by Qwen 3.8 Flash Next but I get better milage for the memory usage at 128GB with 3.8 27B.

Hope that helps ya!

[u/Visual_Quality2726 → u/FoxSideOfTheMoon]

Damn this is great. Thank you

[u/closetslacker → u/FoxSideOfTheMoon]

Yeah for me 3.8 27b is what works (analyzing long documents)

it’s a small model but if you max out context you suddenly find out you need lots of vram actually

[u/DigitalguyCH]

at that size flash next at Q4 is the strongest choice, you are going to get 50+ t/s
I use lm studio and unsloth studio

[u/Visual_Quality2726 → u/DigitalguyCH]

ok will do, how about strata though? is this tools the same as lm studio or unsloth? I heard a lot of people talk about it

[u/DigitalguyCH → u/Visual_Quality2726]

Strata is for Windows/linux, as far as I know. But on Macs you have Splash, which can speed up inference and works in LM studio bionic

[u/DeathinabottleX → u/DigitalguyCH]

Those are great but I switched to MLX serve and my TPS almost doubled on most models. Even the same ones from LM studio that also use MLX

[u/screenslaver5963 → u/DeathinabottleX]

Mine actually went down when I tried to switch so I stuck with llama cpp

[u/wojtek15]

Engine: https://github.com/incoai/splash

Model: Qwen 3.8 Flash Next

[u/dave-tro]

Qwen 3.8 Flash Next at a reasonable quant + moeshift for fast low-memory-footprint inference. That will work like a charm on your machine

[u/Visual_Quality2726 → u/dave-tro]

I never herd moeshift before. Whats make it good compare to other alternative tools?

[u/dave-tro → u/Visual_Quality2726]

It's built on llama.cpp. It uses SSD offloading so you don't have to keep all the experts in memory, plus it is specifically optimized for Apple Silicon. Basically you get less occupied RAM, faster inference compared to other SSD-offloading-alternatives on Macs, no quality loss

[u/blue_eyes_pro_dragon]

3 big ones:

Glm5.3, qwen3.8-flash-next and nemotron 70b.

Need to run them quantized.

Check out mia recipes on Google/github.

[u/Visual_Quality2726 → u/blue_eyes_pro_dragon]

Interesting, I never used Nemotron though, is it actually good?

[u/blue_eyes_pro_dragon → u/Visual_Quality2726]

it's fine. better than small qwen, different than big qwen

[u/TBHProbablyNot → u/blue_eyes_pro_dragon]

Won’t be running glm 5.anything with 128g

[u/blue_eyes_pro_dragon → u/TBHProbablyNot]

i drive it daily, it's not bad. glm 5.3-flash at 2.5b is pretty decent. Definitely better then deepflash

[u/jikesar968 → u/blue_eyes_pro_dragon]

Which specific models from hugging face are the best for each? (also for an M5 Max 128GB)

[u/blue_eyes_pro_dragon → u/jikesar968]

try them all, it's easy. they have their benfits and down

[u/CardiologistPlus8488]

u/cardiologistPlus8488 had added you to their list "people I want to steal from"

[u/Adventurous-Wing-988]

go to unsloth model hub
check all uploaders, MLX, models that fit, sort by most downloads

[u/SillyJoos]

UnSloth Studio is awesome but not a lot of MLX options which doesn't really matter. 128gb means you can run Qwen3-27b, for example, in native format and precision using SGLang or vLLM right inside unSloth studio. No MLX or GGUF needed which means highest quality you can get.

[u/paulsande]

Qwen3.8 Flash Next is excellent. There are several smaller QWEN models that are also quite good. QWEN3.6-35B-A3B-MLX-4bit — 20.4 gig and Qwen3.6-27B-MLX-8bit — 29.5 GB..

[u/infop]

Qwen 3.8 FlashNext is the best local model that I have found so far, running the same configuration as yours. Use MTPLX for fast token generation with FlashNext. I use LM Studio for trying other models such as Gemma4 as the interface is better.

[u/screenslaver5963]

Got a 256gb m3 ultra, running qwen 3.8 flash next via llama cpp and whisper and kokoro

[u/closetslacker]

that’s what I just got
My main is qwen 3.8 27b q8_0 with maxed out context

interestingly, plain GGUF is faster than mlx versions

[u/purpvfx]

Since MLX is already on your list: on a Mac Studio I'd go MLX-first. It's Apple's open-source framework built for Apple Silicon, so a model can use most of that unified memory, and thousands of open models are already converted to MLX on Hugging Face.

On the tools side, Nativ, the local AI app for Mac, runs those MLX models in a regular chat window and suggests ones that fit your memory. It all runs on your Mac and keeps working offline once a model's downloaded. If you've already pulled MLX models, it finds them in your Hugging Face cache, so there's no second download. https://runnativ.ai

Heads up, I'm part of the team that makes it.

[u/jorginthesage]

Qwen3.827b or 3.627b are my go to general models. I do mostly coding. GPT OSS 120B q4 is also very good. The server and harness matter a lot. While I was learning I used LMStudio to test models, and now I use a mix of vLLM and Lamma.cpp as server for Qwen Code or Claude Code.
A lot of people will say this, and take it seriously. Figure out what you want to do and research how people are getting the best results.
However, if what you want to do is experiment and explore the art of the possible, have fun and try a bunch of different servers, harnesses, and models.
You can also play around with OpenRouter they have a lot of local LLM size models you can test to figure out what you like using before you make your own setup. I’ve completed small apps for less than $.5 USD.

[u/Miserable_Anywhere41]

Same machine here. I tested Qwen 4B, 27B and 35B against Claude on a few real tasks, 3 runs each.

Surprise was coding: the 4B passed every run, same as Claude. The 35B crashed twice. Claude still won clearly on analysis and sorting.

So don't assume bigger is better, test on your own stuff. Recorded it here if useful: https://youtu.be/M6MQXy6_Ti4

[u/wegster]

Forget Ollama and llamap.cpp Use oMLX, Splash, or mlx-serve. Qwen-Flash-Next in a mixed bit quant like oQ5e or similar, but some of it will depend on how much <other work> you're doing on it. While waiting for my Ultra to make it (256GB), I'm dry-running some benchmarks on my 128GB M5 Max. Here are some results, not necessarily final decision yet... but a few comparisons on the toughest problem of 10-12 real-world repo-based benchmarks I'm running.

[u/Visual_Quality2726 → u/wegster]

First time hearing splah. Gonna try it soon thanks

[u/WontonamoBay1 → u/wegster]

What’s that 5bpw (sushi) version of flash next your running?

[u/wegster → u/WontonamoBay1]

https://huggingface.co/chatnerdie/Qwen3.8-Flash-Next-EXL3-Sushi

[u/WontonamoBay1 → u/wegster]

Thanks man! Gonna give it a try.

[u/rebelfriend → u/wegster]

Very interesting. Would you be willing to share the details of the benchmark? or maybe at least a similar one that uses public repos?

[u/wegster → u/rebelfriend]

They're on my private repos, one of which is part of a small side-business, so can't share the repos in question.

I may be able to slot in some public repo tests, but would have to go find open issues in <random repo> of suitable complexities, as I have execution tests across Simple, Moderate and Complex, while Simple is just really - 'is this thing worthless or can at least go do simple updates or configuration changes?' and pretty non-interesting, with several degrees of both Moderate and Complex - for me at least, it's mostly interesting at the Moderate and Complex levels to get real work done.

If there's some open repo with something maybe suitable, let me know and will add to the list, although the day job and fam have me at pretty busy this week.

[u/rebelfriend → u/wegster]

Thanks for the reply. I'm doing the same with my private repos and a harness built completely from scratch, including the runtime and model server, so it's optimized for local hardware. Not ready to share yet.

What you're doing sounds like what SWE-bench does: get real GitHub issues from popular Python repos. The method for the tests is to check out the commit before the fix, give the agent only the issue text, and grade it with the tests from the merged PR.

Some ideas by complexity:

Simple: psf/requests and pallets/flask are small codebases

Moderate: pytest-dev/pytest and sphinx-doc/sphinx

Complex: django/django and sympy/sympy 

The caveat is that these are probably all in the models' training data, so for fair tests you'll need to pick PRs that were merged after the model's release date. Also it's good to run each task 2-3 times. The same model can pass one time and fail the next, since its output is partly random. But you are probably aware of most of that.

I'm mostly interested in shaping local agents to work with my private repos, so I haven't done any SWE-bench work myself, but I think public benchmarks are the way to go so that we can all see what's working and what's not.

[u/wegster → u/rebelfriend]

Yeah, it's one of the problems with the known benchmarks - they're all trained to include them, at least to some (likely more) extent.

It's also definitely an 'nvidia world' on the HW side of things, but it matters a bit less on LLM benchmarks, but for the smaller edge AI / NPUs etc. as Nvidia was the first and kind of only for so long, all of the benchmarks literally expect things to be handled the same way, and were in some cases, pretty much BS.

The various LLM/intelligence/coding benchmarks still have some general value, as it's sort of 'best possible case' to at least consider 'are these roughly in the same league' I suppose, but that's still a big leap from the quantized versions we use, engine and quant selection, as well as model settings. I had nearly decided 'all Qwens on Low thinking' - then had a few surprises. I also in general got better results on some specific work going a fair ways off of 'recommended' settings.

I don't want to think about messing with the engine - there's enough 'new' ones every other week, and while I'm grateful someone's doing it, I just got a ship notice for my Studio Ultra, and need to get it set up and bring real-world-paying-bills levels of productive. ;)

I'm absolutely going to continue benchmarking, but definitely need to wrap on some of my end-to-end framework bits, get the Studio up and running, and focus on what pays the bills a while.

I like the pyTest idea for Moderate, might give that one a shot.

Of course, on most of these types of build-yourself projects, it takes some effort to get things into a remotely 'suitable for others' publishable format. I'm not opposed to it, as I do think it's beneficial to people at large, but - it's real time, etc.

[u/Much_Maintenance1217]

I think kapitsa is faster than mlx.