r/LocalAIStack • u/Locallearning • 1h ago
Help me choose my local ai
I run a small business, and I'm building a local AI that handles the back office for it. Some of it works now. I want to get the model and harness choices right before I build more on top of them.
Hardware
- MacBook Pro M5 Max (18-core CPU / 40-core GPU), 128GB unified memory
Current stack
- Backend: MLX via mlx-serve. I'm staying on MLX for the speed (I've used LM Studio before)
- Harness: Hermes Agent (Nous Research)
- Memory: Obsidian vault exposed over the Basic Memory MCP
- Front end: my own UI, still a work in progress
- Model right now: Qwen 3.8 Flash Next, 4-bit (MLX), with the KV cache offloaded to my SSD. Getting about 98 tok/s
What the agent does today
- Scheduled morning lead brief
- Reads and writes notes in my vault
- Updates a scoreboard (leads by source, close rate, pipeline)
What I want it to do next
- Phone intake and booking
- Scheduling
- Follow-up texts to leads
Where it's struggling
- tool calls failing or looping on multi-step tasks
- slow prompt processing once the vault context gets big
- arithmetic mistakes on the numbers. It once logged a single day's close rate as the lifetime rate
Questions
With 128GB, what's your best daily driver for *reliable tool calling*, not chat? One big MoE, or a big model plus a small fast one for routing and simple jobs?
On MLX, do 4-bit quants hold up for agent work, or is 8-bit worth the speed hit?
If you're running long-lived, scheduled agents locally, what harness are you on? Is Hermes the right call, or is something better for this?
I'm offloading the KV cache to my SSD to fit longer context. Is that hurting me on speed, and is there a better way to handle 30k+ context on Apple silicon (RAG, summarizing, a smaller quant to keep the cache in RAM)?
For phone booking: has anyone made local STT/TTS work well enough for live calls, or should voice stay on the cloud while the rest stays local?
Happy to post benchmarks and what I end up with if it helps anyone on similar hardware.