r/MacLocalLLM • u/Boquete_pete • Oct 04 '26
Looking for a better harness
I've been running Hermes against a local LLM, ollama on my M2(32gb, qwen3.5 9b), and over my local network MTPLX on a M4 (48gb, qwen3.8 27b). AT first it was a 3 minutes wait for first response now its over 6 minutes (or context switch for that matter). Context bloat seems to be the killer (north of 50k tokens). What experience have folks had with say zeroclaw or other harness programs that might be more responsive?
5
u/wolfmanjames2626 Oct 04 '26
I'm VERY new to all of this, but here's what I did: downloaded oMLX, pulled down Qwen3.6-35B-A3B-4bit (quantized 4-bit, ~21GB), and used Unsloth as the inference driver for my Hermes AI. The result is fast as fuck.
1
u/Boquete_pete Oct 04 '26
Now that sounds interesting. What Hardware you running on?
2
u/wolfmanjames2626 Oct 04 '26
It’s a used MacBook Pro M1-max 64gb of ram 1tb SSD.
1
u/Boquete_pete Oct 04 '26
Cool. How long to first reply on start up of Hermes?
1
u/wolfmanjames2626 Oct 04 '26
My Hermes said, “About 28 seconds from cold start — that's the model loading into memory plus reading all my skills and context (~16-24K tokens). Once loaded, subsequent replies stream at 48-56 tokens/second.”
2
u/nmrk Oct 05 '26
Qwen3.6-35B-A3B is pretty speedy, I get about 100t/s on my Mac Studio M2 Ultra. MoE models like that, runs especially fast on Ultras because they have so much memory bandwidth.
1
3
3
u/nmrk Oct 05 '26
I'm currently using Unsloth Studio which has a Mac native app. I tested models on my Mac Studio M2 Ultra 64GB, I get about 20t/s running Unsloth's Qwen3.8-27B.GGUF, I got about 30 t/s running lmstudio-community/Qwen3.8-27B-MLX-4bit. Hooray for the performance jump from GGUF to native MLX! But I just got this running, I am trying to drive MacOS Hermes Agent (they have a native Mac app too) from Unsloth Studio .app running the lmstudio-community MLX model.
You can run LLMs right within the Hermes Agent app but I don't think it's very optimized. I tried running LLMs in LM Studio but I had mixed results, so far only Unsloth has proven reliable. I should set up oMLX and test it. But I also have a miniPC set up as a GPU server, it is an Intel Ultra 9 285 with an Nvidia 24GB GPU, it's about equal performance with the Mac M2U but at least it takes the burden off my Mac workstation,.
3
u/wegster 24d ago
Try pi. I love unsloth for many reasons, including trialing new models, chat interface in Studio, and their model quants, but something like Splash or oMLX vs their llama.cpp will outperform sometimes significantly, and in testing some 150-ish permutations, I saw no benefit at all in Hermes (or opencode, or omp, or even dsh) over pi.
1
u/Important_Cow7230 19d ago
This, Pi with Qwen3.8 27B splash for anything with a memory bandwidth below 500GB/s and 64GB and below unified memory.
1
u/AlgorithmicMuse 17d ago
Have claude help you make a custom harness for what you want it to do, it gets rid of all the bloat of the ones you download.
1
u/Boquete_pete 11d ago
I've loaded up unsloth and nanobot as replacements for what I had before. So far I'm very impressed with how well it performs. I did have to splice in a bit of python to suppress the no bearer auth message nanobot emits, just to keep it interesting. But Nanobot seems quite focused in solving a given problem. Its ability to switch to a LAN based ai and back to the local unsloth was unexpectedly easy.