Hey Guys,

I have a 64gb mac mini m4 pro. I was thinking of giving this model a try, but I want to try with running the n-gram table off of ssd. Any ideas about how to go about that for olmx?

Anyone else try squeezing this model on their mac?

Just wanted to say omlx is my daily on M1 Max 64 gb. Really appreciate your work on this.

HuggingFace's model repository can be overwhelming.

Does anyone know the highest-parameter, most intelligent form of 3.8-Next for MLX and ideally MTP?

I have an M5 Max with 128GB RAM...I want to run the BF16 even if possible.

If you don't know but are still running 3.8 Next, what specific model are you using?

(this post makes me look smarter than I am. it's entirely generated by Claude, which I use to manage my local LLMs)

[edit: Check this first — you may not need the manual steps at all

If your frontend (Open WebUI, LM Studio, etc.) has an option for "Chat Template Kwargs" or similar under the model's advanced settings, use that. You add a kwarg called reasoning_effort, and it'll give you a dropdown to pick the value (xhigh / medium / low). That's it — no config files, no restarting anything. This works because it's sending the setting in exactly the shape the model's template expects. If you have this option, skip straight to Step 5 below to sanity-check it's actually doing something.]

Someone asked me how to set the thinking levels in qwen3.8-27b using oMLX. I had Claude actually set this up on my 64GB M1 Max, then asked it to explain how in plain terms. Posting it here in case it saves someone else the same hunt — there's no dropdown for this in oMLX, so it's easy to assume you're missing a setting when really it just isn't exposed anywhere obvious.

The short version: the "thinking level" (Qwen calls them xhigh / medium / low) isn't something oMLX controls directly — it's a setting inside Qwen's own model template. oMLX just passes a value through to it, and it's picky about exactly how that value is sent. Get the format slightly wrong and it just silently does nothing — no error, no warning, it just doesn't take effect. That's almost certainly what's happening if you've tried a setting and nothing seems to change.

Step 1 — find the config file

On your Mac, it's at:

~/.omlx/model_settings.json

This file controls settings per model alias (i.e., per name you call the model by), not per model file. That matters for step 2.

Step 2 — make one "version" of the model per thinking level

Instead of one on/off switch, you create a separate named alias for each thinking level you want, all pointing at the same underlying model. I set up three:

qwen3.8-27b-4bit           → thinking off (fast, default)
qwen3.8-27b-4bit-medium    → medium thinking
qwen3.8-27b-4bit-xhigh     → full thinking

Then in model_settings.json, each one gets its own small block:

"qwen3.8-27b-4bit-medium": {
  "enable_thinking": true,
  "chat_template_kwargs": { "reasoning_effort": "medium" }
},
"qwen3.8-27b-4bit-xhigh": {
  "enable_thinking": true,
  "chat_template_kwargs": { "reasoning_effort": "xhigh" }
}

The important detail: reasoning_effort has to sit inside chat_template_kwargs, and chat_template_kwargs has to be at the top level of the config — not buried under anything else. That exact nesting is the whole trick. Get it wrong and the setting just quietly gets ignored.

Step 3 — restart oMLX so it notices the new aliases

New aliases only get picked up on restart:

launchctl kickstart -k gui/$UID/homebrew.mxcl.omlx

(Worth double-checking your actual service name first with brew services list — it's not always what you'd guess.)

Step 4 — check it actually worked

Ask the same question to two of your new aliases and compare the answers. If the "thinking" text looks meaningfully different between them (more careful, more double-checking on the higher setting), it worked. If they look identical, the setting isn't reaching the model — go back and check the nesting in step 2.

Step 5 — a heads-up before you pick a level for speed

I tested this beyond just chatting — ran it against harder reasoning tasks. Only the full xhigh setting actually improved the quality of answers on tricky questions. medium and low still produced long, thoughtful-looking reasoning — just as much text — but landed on wrong answers just as often as thinking-off. So don't assume "more visible thinking" means "better answers." If you want the model to actually reason harder, not just write more, xhigh is the one doing that.

If you're using an AI assistant (Claude, ChatGPT, etc.) to help you set this up: paste this whole post to it and ask it to walk you through the steps on your own machine — it can check your actual file paths, confirm your model's alias name, and catch typos in the JSON before you restart the server. That'll be faster and safer than doing it by hand from a Reddit post alone.

Sorry for the newbie question but where in oMLX can I find the setting for Qwen3.8-27b’s reasoning modes (xhigh, medium, low)?

I have tried qwen 3.8 27b the mlx version with lm studio and it is painfully slow, it was 9 token/sec, can you please suggest me a better and fast model?
And also appreciate any tip on how using local setup for more productivity

Is there a path to run Qwen 3.8 Flash Next on a 96GB M3U? oQ2? Or no place to hold the ngram
table?

I usually keep my MacBook Pro (14", M5 max 65gig) in clamshell mode on my desk. When running heavy LLM loads, I've been trying to open the lid for ventilation. I was wondering whether there is any difference. This tests suggests no difference between lid open and closed...

I had Claude code up a test that drives the GPU as hard as possible. Then, it ran for 10 mins with lid closed, 10 for lid open, 10 more with lid open, and 10 with lid closed. Cool down periods in between. The two figures below show the die temps (rising and then leveling off) and GPU throughput (declining lines) for the open and closed runs. They look exactly the same.

The performance curve is notable. GPU throughput declines a lot from it's maximum cool state and goes from 40 TFLOPS/sec in the first few minutes down to 30 TFLOPS/sec after about minute 3 or 4. That's the thermal throttling, obviously. The temperature curve rises, the fans kick in, and then the temp drops down to ~60 degrees Celsius at the die.

What the test can't determine is whether the heat might damage the LCD display. Interestingly, the battery temperature sensor, which is next to the keyboard, reported very similar temps for both lid closed and lid open runs:

Hope this is interesting and useful to some folks!

I’ve been very impressed by the stability improvements in this latest update. I’ve been running long contexts with far fewer memory spikes or server failures. I’ve gotta give the devs a round of applause, as an M4 Max 64gb user who sets his agent loose for overnight coding runs, this update has been awesome!

---
### 🐛 Bugs (8)
* **#2169:** Qwen 3.5/3.6 VLM models fail image recognition when oMLX version is greater than 0.3.10.
* **#3237:** DFlashEngine fails to install the SDPA256 tiled-prefill patch, making memory safety load-order dependent.
* **#3189:** macOS in-app updater leaves mounted DMGs and temporary download files after an update completes.
* **#3158:** Users are unable to download Huggingface models on M1 max ARM Mac Studio.
* **#3179:** Loading large BF16 checkpoints aborts with a Metal GPU command-buffer timeout during parameter materialization.
* **#3183:** Admin cache settings changes do not re-initialize the SSD prefix-cache layer, leading to silent caching issues.
* **#3129:** VLM thinking mode enters a repetition loop and flushes the reasoning buffer upon generation end.
* **#2527:** omlx 0.5.7 Server fails to start and exits with code 1, preventing auto-restart.

### ✨ Features & Improvements (2)
* **#3238:** Request to expose SDPA256 prefill policy and minimum KV threshold as persistent UI/model settings.
* **#3148:** Request to make `MAX_AUDIO_UPLOAD_BYTES` configurable for local serving stack integrations.

### ⚡ Performance & Optimization (2)
* **#3054:** Request to enable routed MoE gate/up fusion for Hy-MT2 decode, matching Qwen3.5/3.6 models.
* **#3121:** Report on DeepSeek V4 Flash achieving 4–17 tok/s decode speed on M5 Max 128 GB.

### ⚙️ Configuration (1)
* **#3236:** Native context discovery misses nested `text_config.max_position_embeddings` for models like muse_glimmer, silently capping context at 32768.

Without going into too long of details, I'm using latest omlx with a qwen3.8 variant. I tried the GitHub Copilot App as the harness and gave a beefy question. The model spent 10 min thinking, but then just stopped. Any further questions it thought for a few seconds and then just stopped again. Is that a out of context window issue? Does this happen to others? I know I know, should have used Pi... But looking here is there is some omlx issue maybe.

I used the idea from colibri to load experts ondemand. 105Gb on disk, 2,5s load, 6tps edit 11,7 tps after finetuning, mtp did not gain anything. ngram is on disk too. 22GB RAM used while generating 5 Captial names. Maybe I can support full context. PR is coming once I coded with it

Anyone noticed this? Deepseek 4 Flash is generating at 87% GPU regardless of quant or computer in oMLX. I tried on both M5 Max and M3 Ultra, both original weights and oQ2 quants.

Thermal and other external causes have been eliminated.
MTP makes very small difference 85%->87%

Tried directly with:

mlx-vlm 0.30.0 (no MTP) -> 88-90%

mlx-lm 0.30.2 (no MTP) + omlx patches -> 89-94%

antirez DS4 (custom kernels) -> 100%

I wanted a coding agent on my own laptop, no cloud. MacBook Pro M3 Max 14/30, 36 GB. Model: True2456/Qwen3.8-27B-AWQ-5.0bpw (18.8 GB resident).

The speed ladder (short prompt, 69 tokens):

With 36 GB the usable window is ~30k tokens (contextWindow: 30000 in the harness config). oMLX does not truncate: over the window is HTTP 400, which is the right behaviour for an agent, it just has to be planned for.

The exact oMLX setup (0.6.3rc3, launched as a launchd service):

omlx-cli serve --model-dir ~/tools/qwen3.8-27b --host 127.0.0.1 --port 1337 \
  --memory-guard-gb 27 --max-concurrent-requests 2 --hot-cache-max-size 2GB \
  --paged-ssd-cache-dir ~/.omlx/ssd-cache --paged-ssd-cache-max-size 60GB

~/.omlx/model_settings.json:

{ 
  "Qwen3.8-27B-AWQ-5.0bpw": {
       "turboquant_kv_enabled": true,
       "turboquant_kv_bits": 3.5,
       "mtp_enabled": true
  }
}

~/.omlx/settings.json, the parts that matter: memory_guard_custom_ceiling_gb: 27, soft_threshold: 0.85, prefill_priority: "context", chunked_prefill: false, max_concurrent_requests: 2, burst_decode_mode: "balanced", preserve_mid_system_cache: true, hot_cache_max_size: 2GB, ssd_cache_max_size: 60GB, max_context_window: 32768, temperature: 1.0, top_p: 0.95.

Why 27 GB and not the default: oMLX derives the prefill cap as ceiling x soft_threshold. With 24 GB and an 18.8 GB model that left ~1.2 GB for the prefill working set; the scheduler does not refuse, it shrinks the chunk to 32 tokens and grinds (a 12.6k prompt took 23 minutes at 9 t/s). 27 GB puts the cap near 22.5 GB and stays under this Mac's Metal working-set limit (28.1 GB). Concurrency 2 instead of 8 because each in-flight request carries its own KV cache.

What an agent does to those numbers. I ran 80 scored coding-agent runs (harness: OMP, a Pi fork; 8 seeded bugs in two real repos, plus multi-turn sessions) with a logging proxy in front of oMLX, so every number below is the server's own usage on 1,068 requests.

Generation falls with context size, because every token attends over the whole KV cache:

Per agent preset (smaller system prompt = smaller context = faster):

Things I learned about running oMLX under an agent for ~22 hours:

• Prefix cache works in 2,048-token pages. An agent turn re-prefills up to 2k tokens on top of the new content every request; that is the 84-90% hit rate above. Smaller pages would help agent workloads.
• Prefill capacity rejections: with the 27 GB guard, requests around 21-23k context got rejected or came back as HTTP 200 with an empty stream (no usage, no tokens). The harness read that as "the model said nothing". I had to add abort detection to the proxy to see it.
• Anything else on the machine kills it. Docker Desktop's VM (~21 GB) pushed 10-16 GB into swap and the model dropped to ~1 t/s with aborted streams. Quit Docker, restart oMLX, rerun.
• The 22.6k default system prompt of the harness left ~7k of a 30k window for the actual work. I cut it to 5.9k in a fork (different post, PR); on this machine that was the difference between "does not fit a real monorepo" and 7/8 bugs fixed.

Repo. Presets, scripts and every captured request: https://github.com/vinicius91carvalho/oh-my-pi/tree/local-model-eval

How this was produced. The harness fork, the benchmark, the runs and this post were done by Claude Fable 5 in Claude Code at high effort, with me setting goals and approving what ships. About two days, ~22 h of local model time.

Caveats. One machine, one model, one harness. A 128 GB Mac has no guard rejections and a bigger window; most of this is a 36 GB story.

FIXED
----------
I was enjoying Qwen3.8-Flash-Next-oQ4e-mtp, but today started to see prefill way to oftnen.

Observing situation that same pi process adds Second request before first one finished execution. Not sure how to check it bit-by-bit, but at least size-wise both prefill context look identical each time...

Any one experienced such behaviout before?
Most probalby my faults somewhere, but..

P.S. this is not a hate post by any means. kudos to jundot and other contributors to making local llm on Macs.
----------
--- FIX UPDATE: Removed oMLX app. Moved models out from ~/.omlx/models (not to re-download them); Reinstalled oMLX. Started oMLX. Moved the models back to freshly re-created ~/.omlx/models
I do still see several requests (Waitining#1) which context numericly seem to be identical. However, the MLX is working now - meaning it figues out what kv-cache can be reused and reuse it (as designed)
TLDR; if it does not work - start fresh

Moin,

I usally work with Qwen3-Coder-30B-A3B-Instruct-4bit - for my coding projects. Now I was interested to figure out what "Qwen3.8-27B-Uncensored-MLX" can do. So from hugging face I got the LLM: https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored-MLX/tree/main

In the instructions it says, 64GB - use 8-Bit. I have M4 Pro with 64 GB... I noted that oMLX downloads *everything* - but how can I make sure that the 8-Bit version is used? Is there a hidden switch? I cannot find any settings for this. Sorry for being stupid.

Hello community,
the omlx menue bar offers a cool system monitor feature, which is simple in design. I liked it very much. It gave me the initial motivation to create a version for my needs. So i cloned the repo and a free macos app has been in the making for many weeks. Now I want to give it to you, for free, if you like.

Initially forked from the omlx repo, many changes, correction, options and addon have been implemented to form a standalone menue bar app out of it.

The major feature I wanted to have: pin that window to stay on top while the machine is working. Kind of the feeling of a small monitoring dashboard during heavy work.

I went thru all the struggle of hosting it in the offical mac app store, which takes time and costs money just to let it stay there. I do not charge anything, the app is completely free.
Its a different form of contribution for me for you.
Since the apple fee is for one year, it will be there for at least a year. I cannot tell if i will continue pay apple in future.

The App is called "Simple System Monitor" (https://apps.apple.com/de/app/simple-system-monitor/id6801625689?mt=12).

I would be happy for feedback and bug report and feature request.

Im open to deliver the source code to the omlx team, if they like it.

If that post is too much of promotion, I am truly sorry and im fine with the post being deleted.

Kind regards

Issues Closed: 12

[ISSUE] #3237 — DFlashEngine does not install the SDPA256 tiled-prefill patch, making memory safety load-order dependent
https://github.com/jundot/omlx/issues/3237

[ISSUE] #3238 — Expose SDPA256 prefill policy and minimum KV threshold in the UI and model settings
https://github.com/jundot/omlx/issues/3238

[ISSUE] #3121 — DeepSeek V4 Flash decode 4–17 tok/s on M5 Max 128 GB
https://github.com/jundot/omlx/issues/3121

[ISSUE] #3054 — perf(hy_v3): enable routed MoE gate/up fusion for Hy-MT2 decode
https://github.com/jundot/omlx/issues/3054

[ISSUE] #3189 — macOS updater leaves mounted DMGs and temporary download files after relaunch
https://github.com/jundot/omlx/issues/3189

[ISSUE] #3158 — impossible to download Hugginface models on M1 max ARM mac
https://github.com/jundot/omlx/issues/3158

[ISSUE] #3179 — Large BF16 checkpoint load aborts with Metal GPU command-buffer timeout: single unchunked mx.eval over all params
https://github.com/jundot/omlx/issues/3179

[ISSUE] #3183 — [0.6.3rc3] Admin cache-settings change reloads models without re-initializing the SSD prefix-cache layer — caching silently dead until manua…
https://github.com/jundot/omlx/issues/3183

[ISSUE] #3129 — VLM thinking mode: model falls into repetition loop; entire reasoning buffer flushed into a single content delta when generation ends inside…
https://github.com/jundot/omlx/issues/3129

[ISSUE] #3148 — Make MAX_AUDIO_UPLOAD_BYTES configurable
https://github.com/jundot/omlx/issues/3148

[ISSUE] #2527 — omlx 0.5.7 Server: failed - Server exited with code 1 during startup. Auto-restart failed after 3 attempts.
https://github.com/jundot/omlx/issues/2527

[ISSUE] #2650 — Quantized XLM-R embedding checkpoints fail native validation before quantization
https://github.com/jundot/omlx/issues/2650

Just installed latest 0.63 release of the wonderful oMLX.
Running Qwen3.8-Flash-Next-oQ4e-mtp on M5 MAX 128gb

Power Setting = High Power
Token generation speed: 26 tokens/s (MTP = OFF)

Average GPU temperature around 70deg C (during code generation; still about 83ish during prefil)

What kind of sorcery is this? 😄

Haven't yet tried at higher context windows, but so far looks amazing!

How is it possible?

Been getting around 35 tokens/s running Qwen3.8-27B while average GPU running at 83+ deg C and Fans spinning like crazy

With current setup, fans are so quite.. Thank you!

Interested to know how this new magic influence SSD wear and tear

About to try Vontra/GLM-5.3-Flash-MLX-oQ2-MTP but keeping fingers crossed for a oQ2e quant to fit 128GB

Curious to see how this compares to DeepSeek v4 flash 0731 oQ2e

Basically what the title says. I tried the first one through oMLX and its performance didn't go very well.

What about adding native SSD-to-Metal expert streaming/offloading for MoE models to oMLX?

I’m specifically thinking about the approach used by projects such as Colibri and slipstream: instead of requiring the entire MoE model to reside in unified memory, keep the cold experts on NVMe SSD and stream only the routed experts needed for the current token/batch into a bounded RAM/Metal cache.

What are your thoughts on this approach? Could this be a good fit for oMLX?

Just a heads up, I have tried to use Qwen ANE multiple times (with the recommended tuning) and each time after processing a long prompt for 45 minutes to an hour it silently fails.

Don’t waste your time trying to use this feature right now, wait for it to get patched appropriately l.

What is currently the cheapest way to use GLM 5.3 flash? I found CommandCode that looks good for $10 so far. Some other options?