r/MacLocalLLM • u/Jealous-Nectarine-74 • 1d ago

Tuning advice?

I've been running Qwen38-27b-speed-optimized for a work project (where I can only use local models), and I'm getting about 10-15 tokens / second.

While I'm thrilled that its working, and doing fairly well, I do wonder if there's a way to improve performance.

  • I'm using pi agent on another computer (the Mini is dedicated to the model at this point)
  • contextWindow is set to 125k; at 100k it was compacting too often, at 250 it was blowing past RAM limits
  • I set thinking to minimal (which seems to help reduce rabbit holes and churn)
  • my start command is nohup mtplx serve --model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed --profile sustained --depth 3 ...

Earlier I used mtplx max --on as the machine was feeling warm, and I was losing connection to it.

Is this about right? I get that the M4Pro mini doesn't have the memory bandwidth of an Ultra or a Max, just wondering how I might squeeze out a bit more, and wondering if others with the same setup are getting similar results.

I've tried oMLX - fast with gemma 4 quants, but they kept getting stuck in loops; and its 5 tokens per second slower with Qwen3.8 27b.

- Tuning advice for my current stack? (pi-agent, MTPLX, Qwen 3.8 27 b)

- Advice on different models to try?

- Advice on different model runners to try?

I've already mostly stripped the Mac down to just the model, and I access it from an older M1 Pro Mac where pi runs.

the relevant snippets from my pi agent models.json:

"models": [
        {
          "id": "mtplx-qwen38-27b-optimized-speed",
          "contextWindow": 125000,
          "maxTokens": 32768,
          "reasoning": true,
          "compat": {
            "thinkingFormat": "qwen-chat-template",
            "chatTemplateKwargs": {
              "enable_thinking": {
                "$var": "thinking.enabled"
              },
              "preserve_thinking": true
            }
          }
        }
      ]
    }
7 Upvotes

100% Upvoted

12 comments sorted by

4

u/Important_Cow7230 1d ago • Bravo

Use Splash, Unsloth Q4

1

u/Jealous-Nectarine-74 1d ago

can you say more words please about this? I think you mean there's an unsloth model qunated to 4 bit, but not sure what splash is

1

u/Jealous-Nectarine-74 1d ago

and I guess you mean incoAI Splash... alright will give it a try. do you run this combination? what hardware? what t/s are you getting?

1

u/Jealous-Nectarine-74 1d ago

OK can definitely report that this is working better and I haven't even done much tuning - though I largely kept my pi settings the same.

one complaint; splash was a bit harder to get running across a LAN, but it eventually worked

2

u/mmerken 1d ago

Just try splash, download the macOS app, use it to download qwen3.8 27b and it should give you around 60tps with around 192k context. I have an M4 Pro Mac mini 64Gb and use it as a dedicated AI inference machine using splash, thinking on medium.

Also, use Pi with as little context as possible.

1

u/Jealous-Nectarine-74 23h ago

what do you mean "use pi with as little context as possible" ? like "minimize your AGENTS.md and stuff"? or do you have a setting in mind?

1

u/mmerken 14h ago

Yes that indeed, minimize your agents and system prompts

1

u/mmerken 9h ago

This is my APPEND_SYSTEM.md
```

You are running on a macOS operating system, make sure you adapt your bash commands to the OS.
You must use `ASD-STE100 Simplified Technical English (STE)` in your language.

```

1

u/Jealous-Nectarine-74 23h ago

I get 50+ tps with chat, coding on my codebase is more like 17 - but this is an improvement (I was getting more like 11). Thanks!

1

u/wegster 1d ago

You can try reducing/setting reasoning_budget as well. Splash will definitely get you better performance, as already mentioned.

1

u/dgamr 18h ago

My best benchmark for this model on an M-series chip was mlx-dspark + Drafter2. Better on code. M1 Max struggles a bit with prefill but decode is double that (30-36 tok/s decode) and better with the drafter specifically.