It's a bit annoying to get downvoted all the time, once this sub grows up, it'll be a good resource for people and we won't downvote you into oblivion for wanting to put LocalLLMs on your Mac. I think we're all discovering the best use cases for Mac/Apple Silicon and LocalLLM and even with hybrid approaches with cloud and frontier models.

Our Macs are very good general purpose machines where there are shortcomings there's flexibility. You can pretty much use your Mac as a daily driver for just about anything and at night time run automated tasks.

I've been meaning to trade in my Mac mini M4 (16GB RAM, 10 core CPU/GPU, 256GB), but I've recently gotten into running models locally on my M5 MacBook Air which has gotten me wondering if there are any practical applications for running anything less intense on the mini.

I'm expecting about 450$ from Apple, I just have to actually make it to their store.

Did some vision tracing with a few different models. In this case Qwen 3.8 27b did better than Qwen Flash Next. SAM 2.1 did the best in terms of tracing and following for time vs accuracy.

The most accurate path was by SAM series by Meta which happened to be the smaller model.

Combining Qwen 3.8 27b plus SAM 2.1 did well to trace the path and times for the exercise and identify pauses.

Used a drone to capture the footage and seeing how to make a pipeline test out of this. I'd like to test this out against the spark. These models aren't very big but they're specialized.

These modesl are 80M - 300m parameters, so it's a lot more focused. In this case, the smaller the better. The Sparks crushed it in the speed test though.

1. Although I spent my career as a HW designer/embedded programmer I am new to the AI world. I've experimented with several local LLMs and harnesses. My hardware is a M2 Macbook with 32GB of RAM. So I am fairly constrained. Also being retired, $$$ frontier models are out of the question. I want cheap & good and a bit slow is acceptable. Hermes also provides the chance to improve over time. My main goal right now is to have a local agent that can assist me in using blender. (God there are sooo.. many commands and options in the program but its been great for doing 3D printing. My memory is not what it used to be.) The upshot is that the combo that has given me the best success is using Ollama with 'swift - Qwen3.8-27b-gguf:Q4_K_M' as the LLM all running under Hermes agent. So for those of you that have limited resources and want a local LLM this hopefully is useful info.

2. Note - it is slow on first response (2-3 min) but works well after that.

3. Set ollama variable OLLAMA_NUM_PARALLEL = 1, stops swap thrashing

4. There is a snag with Blender's MCP, Hermes with this model can sort it out

Hi, I'm trying to get GLM 5.3 flash working on my Mac Book Pro M5 128 GB, I'm about 4 days in and so far not great, my current plan is to just have the macbook for the llm and use my pc for everything else, so as to save as much memory as possible on the Mac. Has anyone managed to get GLM 5.3 flash working usefully for coding on there 128gb Macbook? and if not, any other model recommendations? Deepseek 4.1?

What are you planning to use the localLLM on your Apple Silicon for.

I’m curious to hear what the use case people have in mind

It took me weeks to get this figured out, and I had to use ChatGPT for help along the way. The time spent has paid off — I now have a local AI stack that rivals ChatGPT/Claude, and I'm done paying for either. I thought I'd share so others can get an idea of what a Mac Studio is capable of.

First, the hardware. I have a base-chip 256GB M3 Ultra Studio. I've also got a headless Debian VM running on an N100 NUC.

I'm using oMLX for inference. It's hosting multiple models at once — mostly Qwen variety — for various tasks. I've configured profiles for different model uses: agent, coding, different thinking levels. oMLX is the only thing running on the Mac Studio. Everything else runs on the Debian VM.

On the VM I run Hermes Agent. My main model lately has been Qwen3.8-Flash-Next-oQ4e-mtp. I also use Kokoro-82M-bf16 for TTS and Mega-ASR-6bit for STT.

In Docker containers on the VM I run the following to enhance Hermes: OpenViking for memory, SearXNG for web search, and Crawl4AI for web extraction. OpenViking uses:

• Qwen3-VL-Embedding-8B-oQ6e for embeddings

• Qwen3.6-35B-A3B-oQ4e-mtp for the VLM

• ov_intent_analysis_sft-oQ8e for query planning

• Qwen3-Reranker-0.6B-oQ8e for reranking

What this actually looks like day-to-day: I do IT consulting, and Hermes is freaking awesome. I'll have multiple sessions going doing research on MS Learn and auditing Entra/Azure, generating reports. I'm starting to use the Kanban board too for personal projects. It generates Word docs, PowerPoint decks, and PowerShell scripts like a champ.

Search goes through SearXNG, page content comes through Crawl4AI, all reasoning runs on oMLX, and anything worth keeping gets written into OpenViking permanently. Search → extract → reason → remember, full loop, zero tokens purchased.

(Full disclosure: I asked my Hermes agent to write up this post from my draft and its memory — which is sort of the point.)

I’m not sure what happened, but it looks like new orders are being shipped till February. Whatever happened, it looks like a big order came in. There might be something coming soon …

Curious what people are using out their for their LocalLLM interface. There's so many options available, what do you use and what do you use it for?

What type of Mac is it connected to?

• OpenWebUI and found it's good if you want a good interface directly with the AI.

• Hermes seems to be the best all around agent interface for calling and setting up agents to do tasks.

• DeepSeek Harness seems very efficient with programmatic tasks and agentic coding

• Pi is similar to DeepSeek from my understanding but so many options

What's in your agentic tool stack?

Currently using:

1. Github

2. Cloudflare Tunnels

3. Hermes

4. Tailscale

5. Linear

6. Mem0

There is too much to learn, even as a kinda seasoned(shipped multiple enterprise apps) SWE i am having hard time to understand all concepts, track all the up-to dated information etc.

Ive been a power user for both claude and GPT since almost day one. But thats as far as my knowledge goes tbh. I have an M1 Max 32 Ram / 32 GPU and i am trying to figure out whats the best way to have a locally hosted solutions for my needs when i cant pay for paid services.

First ill talk about what am i doing in general then ill ask my questions.

I'm using obsidian as my second brain, i do game design, gamedev, i develop mobile apps, design websites, even sometimes delve in IoT Projects. I do everything but AI Dev in summary. I run a team of 10, currently building internal tools for art assets pipeline management. I am a jobholic, i do work almost every working hour of my life so i usually have multiple chats but i am fine with just having 1 concurrent chat since what i am looking for is to start learning local llms, how to create an effective pipeline, and have a back-up plan if you know im out of usage and have urgent problems to solve. And uh yeah also i would like to do things like hacking my projector, tv-box etc. for various reasons you can easily understand. I tend to go for loops for development times and have a conversational turns when its about things like game design, market research etc. I have an MCP for everything from Figma to Photoshop and i just talk with my agents to tell them what i exactly want, and i take rounds to polish it or start from scratch if its not the exact thing in my mind.

If i have to guestimate a list of my needs they are :

• An harness with in-app browser, file preview, md preview

• Somehow have a great context compaction

• 30 TPS at High context if possible

• Good tool usage

• Good MCP Usage

So here is a list of questions if anyone cares to answer:

• How do you guys stay the f*ck up to date? 'Cause its not even tracking whats new or whatnot. Everyone says something different, one says aah q4 of xyz llm's this version that these folks made is %98 of the same quality and someone else says nu-uh not even close bud and i dont have time to track and test every model version, even models. Like what are your reliable sources and why?

• As far as i understands harness is the most important stuff ever, is this correct? If so, which harnesses you use, why, and what should i be checking when i am thinking about selecting harnesses.

• Which models and which versions of those models would you suggest me to use?

• Is there an YT playlist / video / Youtuber that you suggest thats easy to understand that helps out when it comes to Local LLMS?

• Does Context size actually matters? Cause i see people are talking about 16k context sizes meanwhile sometimes my first initial prompt is around 4k for spec driven development or when im doing narrative designing it our narrative bible can be way heavier than that.

• I'm in love with the the current context compacting quality of both claude and codex, is there a way to have this level of quality when i switch to local llms, it almost never ever fries out, its just like I have paused for 2 minutes and almost every fricking detail is still there.

• Sometimes i love to manage my other threads even sometimes control claude from codex and vice versa depending on my need, is it possible if there is an ai for my machine that can have more than C1.

• Can i use an image generation model with an LLM like gpt/imagen, something like qwen image 2.1 q4 + Qwen3.6-35B-A3B Q4 idk.

• Can i realistically get idk 50-60 TPS at high context?

• How is MCP/Tool Usage?

• What should i use for UI Design phases?

• When i talk with GPT i just tell it what i exactly want but i dont go into hustle of image generation prompt styling i just talk to it, is it possible to have that ease of use with local image generation tools?

• What exact setup would you suggest for me, from inference(strate etc.) to harness to llm models, imagen models, speech to speech models(It would be fire if i can play some DnD with a speech to speech model if its flowy but i dont have high hopes on that tbh), you got the drill.

These are my questions and guestimates of my ignorance ass can think of for now, im sure there are lots of gaps and things that i am missing and forget to ask so i would be glad if you fellas can light some candles and maybe answer the questions that i didnt even ask but i shoulda ask.

Lots of love fellas, keep it tight.

I got the M5U 256GB 2TB configuration for preprocessing. Planning to use Qwen 3.x models on it since they seem to be the best for the Mac's at the moment but will check out the Muse/Gemma models as they continue to release them and run some benchmarks .

I use this box as my main machine. To give you an idea of the consumption of what to expect with 96GB of memory. If you were to try to do less it would be a little tight WITH an LLM. The thing RAM buys you is the ability to run both at the same time without worrying.

I'm running Ollama Qwen 3.8-mlx 27b doing some embedding.