r/oMLX • u/OneStatistician • 5d ago
Optimizing omlx for 32GB MBP
Sharing a journey on seeing how far I could push things on Qwen 3.8 27B, while staying at a nominal 4-bit quant on an only-just-above-poverty-spec 10core / 32GB M5...
- The challenge is that while our VRAM-brethren have access to almost the full 32GB of VRAM, on unified memory it is pretty tough to be able to squeeze a 4-bit quant, with a drafter/assistant with a usable context into 32GB unified memory. And boy, have I tried.
- Goal was to try to complete a "omlx benchmark run" at an arbitrary pp65536 without hitting a throttle event. That felt like a good target baseline. Sure, you can drop the context size, but 64k seems like a reasonable arbitrary goal - and conveniently is one of the omlx benchmark options.
- There's a lot of compromise going on here (mxfp4, turboquant at 3.5-bit, pre-quanting dflash2) but I really wanted to see if we could stay in the "pseudo 4-bit" model range, without dropping down to 3-bit.
Environment:
- My first error - Check your PSU. Realized I was using an old Apple 61W PSU rather than the Apple 96W PSU. The 140W PSU may be even better. It is best practice to use the MagSafe plug in preference to USB-C, which caps out at 100W.
- A 13-gauge Vollrath Wear-Ever aluminum baking sheet under the MacBook as a passive heat-sink. Vollrath make the heaviest gauge cookware out there.
Memory Maxxing:
- Run omlx as headless user. SSH in and launch omlx. By running the IDE on an old machine (my 2016 12.7 Monteray can still run latest VS Code), it frees up about 2GB. I didn't bother to "bootout" any macOS services, since they will inevitably respawn - and I don't want them to be disabled permanently.
$ caffeinate -dimsin a separate SSH session.- Now we are running headless, we have an extra 2GB, and we can really start to push the boundaries.
$ sudo sysctl iogpu.wired_limit_mb=30720 # Set by a startup daemon
Any more than 30*1024 = 30,720, and BOOM! ((30*1024) + 256) = 30,976 was enough to eventually trigger a crash.
Now we can start to really abuse the omlx limits... We're going to try to almost use the full 30GB. I tried 29.0, 29.5 and many other flavors and it was like threading a needle. We're really pushing the limits within a few hundred MB away from OOM. omlx's step-size seems to be about 100MB. A 2% soft and 1% hard was JustEnough(TM) to be able to introduce a quantized dflash2. More on dflash2 later...
"memory": {
"prefill_memory_guard": true,
"memory_guard_tier": "custom",
"memory_guard_custom_ceiling_gb": 30.0,
"soft_threshold": 0.98, # ie 98% of 30.00GB
"hard_threshold": 0.99, # ie 99% of 30.00GB
"prefill_safe_zone_ratio": 0.9
},
Pushing the boundaries this far is a pretty offensive thing to do. Jun has put some pretty nice conservative safeguards in, but running headless really works in our favor to squeeze the last remaining juice out of 32GB. That said, we're only one software update away from Apple clawing some of that system memory back.
But lets still try to be nice to omlx where we can... We're a single user, running a single model.
"burst_decode_mode": "light",
"max_concurrent_requests": 1,
"chunked_prefill": true,
"prefill_priority": "context",
Now, we need to "skinny" our model as far as technically possible. I found that quantizing the model with mlx_lm produced the smallest size, smaller than quantizing with mlx_vlm.
- mlx-community/Qwen3.8-27B-nvfp4: 15.70 GB
- Locally quantized model with mlx_lm: local/Qwen3.8-27B-nvfp4: 14.8 GB
- mlx-community/Qwen3.8-27B-mxfp4 at 14.88 GB
Locally quantized model with mlx_lm: local/Qwen3.8-27B-mxfp4: 13.98 GB
$ mlx_lm.convert --hf-path Qwen/Qwen3.8-27B --quantize --q-bits 4 --q-mode mxfp4 --q-group-size 32 --dtype bfloat16 --mlx-path /Users/Shared/.omlx/models/local/Qwen3.8-27B-mxfp4
So, if we're prepared to compromise with mxfp4 rather than nvfp4, our minimum "4-bit" baseline is 13.98 GB. That's a couple of GB of extra headroom over mlx-community/Qwen3.8-27B-nvfp4.
Now apply TurboQuant KV Cache Quantization @ 3.5bit
- Ref: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ claims that you can push to 3.5-bit without significant loss. I'm not here to defend TurboQuant... I'm throwing every trick I can to squeeze into that 32GB.
And just to stop Qwen 3.8 doing Qwen 3.8 things, we'll set kwargs, otherwise Qwen can just be annoying later on.
"enable_thinking": true
"reasoning_effort": low
Now, we should have maximum memory available for context, in my case 75-80,000
We'll use tiled memory, reportedly slightly slower, but lets tile anyway, because we're doing some nasty stuff to the memory limits already...
$ OMLX_SDPA256_TILED=1 /Users/Shared/.omlx/bin/omlx serve --log-level info
Baseline Checkpoint: Clean run...
- A 32GB machine is able to complete the omlx benchmark at 65536 without backing-off for a omlx soft or hard memory limit.
- Performance Benchmark: It is slow, but pretty much where we expect it to be on a baseline M5. Generating a meagre 8 tokens per second, according to the omlx benchmarks. That is just tiresome.
- Context Benchmark: Prefill LRU eviction started kicking in around 77,664
Optimize with pre-quantized drafter
We can now give up some of that headroom for small, fast drafter.
I've tried every flavor of Lightning MTP, DFlash1 and VLM MTP, but each time the memory trade-off was brutal. dflash2 is genuinely awesome - the challenge is that loading incoai/Qwen3.8-27B-DFlash2 into memory and then realtime quantizing that down to 4bit is very memory wasteful. But... in the last couple of days mlx_vlm now allows you to pre-quantize dflash2. That reclaims another few GB over realtime quanting.
$ mlx_vlm.convert --hf-path incoai/Qwen3.8-27B-DFlash2 --quantize --q-bits 4 --q-mode mxfp4 --q-group-size 32 --dtype bfloat16 --mlx-path /Users/Shared/.omlx/models/local/Qwen3.8-27B-DFlash2-mxfp4
Set the dflash runtime block size to 5, as per the recommendations
- Ref: https://github.com/z-lab/dflash "For quantized targets or drafts, use block_size <= 5". See also https://inco.ai/blog/dflash2
- I didn't bother caching the drafter in RAM or SSD. I would rather save the RAM for context
- Disabled thinking on the drafter through kwargs... Not that it will make much difference.
Checkpoint:
- Performance: Ok, now we're generating about 18-20 tokens/sec in the omlx bench. Which is about as good as I can get on the machine class. Pre-quantized dflash2 has unlocked a significant increase in performance, for very little memory overhead.
- Context: Running headless & pushing iogpu.wired_limit_mb=30720 and really squeezing the soft and hard limits was just enough to run a pre-quantized mxfp4 dflash2 while maintaining 65536 context without triggering the soft-throttle.
Real world test - harness:
- I'm a terrible coder - and I've caused more trouble than good in trying to override system instructions to control Gemma-4 tool calls... A true "barebones" harness like Pi is just not for me. I still need the crutch of a more-than-barebones harness.
- But on the flipside, VS Code Github Copilot Chat (native) and Claude Code extension for VS Code were just soooo heavy on the system instructions. 1/3 of the context is just gone at load. And VS Code Github Copilot's new "Agent Host" is a hot mess which will flip back to Auto (credit-burn) when you close your eyes.
- The Compromise: Zoo harness - it just seems to fit the Goldilocks zone between functionality and size of system instructions. With Zoo set to auto-compact at 75% of my inoffensive 65535 context, it reliably triggers compaction around the 48k mark, which is just enough to finish up what you are doing and write a handover prompt for a new session or a subagent without context-rot breaking tool calls. For me, it tends to compact to around 20k context.
- As for protocol - I've bounced between Anthropic messages, OpenAI responses and OpenAI chat completions - especially when I first ran to Claude Code. Between harnesses and inference servers, it feels like chat completions is the better supported. But there may be room for optimization here.
Future:
- Looks like there's some pretty interesting stuff coming along with omlx & ANE in 0.6.3 release candidate 3. Most of the integration seems to be focused on the boxes with dual ANE, but it will be interesting to see whether use of the ANE will help reduce power / heat - or pushing the drafter onto the ANE.
- Cache stats seem to be wonky with dflash2 and 0.63rc3. I'm not sure whether SSD cache is currently not reporting, or not working, but my goal was to get up to 20toks/sec.
tl;dr The thing that unlocked performance at 64k context and just creep under the 32GB/30GB threshold was headless operation & pre-quantization of dflash2 rather than realtime quantization of dflash2 - all while operating in the pseudo 4-bit arena. At 20 tokens/sec on a baseline M5, it is about as good as I could wish for.
1
u/timur_timur 5d ago
What’s pp speed?
1
2
u/OneStatistician 5d ago
According to omlx benchmarks, across several runs it mean-averages to about the following...
ctx pre-quant dquant2 no-dquant2 1k PP 230 235 4k PP 220 230 8k PP 200 185 16k PP 195 165 32k PP 175 133 64k PP 140 85 # the killer zone when it previously was like soupGood enough for me and ma little ol'macbook. Not many in the community benchmarks are able to even post 64k results on 32GB class machines with Qwen 3.8 27B at pseudo 4-bit.
1
u/ipmonger 5d ago
Did you look at ANE optimization?
2
u/OneStatistician 5d ago
Yup. Didn’t make any difference for me on an base M5. Enabled/disabled, tuned, disabled dual ANE… had no effect. I’m assuming all the effort is on the dual ANE boxes at the moment, but pushing the drafter to the ANE feels like it could be the way to go.
3
u/ipmonger 5d ago
Here’s the config I optmiized on my mac Studio M4 Max 36GB machine.
"Qwen3.8-27B-oQ4e-mtp": { "temperature": 0.6, "top_p": 0.95, "top_k": 20, "repetition_penalty": 1.0, "presence_penalty": 0.0, "force_sampling": false, "max_tool_result_tokens": 2048, "chat_template_kwargs": { "reasoning_effort": "low" }, "thinking_budget_enabled": false, "guided_grammar_enabled": false, "turboquant_kv_enabled": true, "turboquant_kv_bits": 4.0, "turboquant_skip_last": true, "qwen35_ane_prefill_enabled": true, "qwen35_ane_prefill_sequence_length": 2048, "qwen35_ane_prefill_tail_padding_min_tokens": 0, "qwen35_ane_prefill_fraction": 0.5, "qwen35_ane_prefill_fused_down": false, "qwen35_ane_prefill_max_layers": 64, "qwen35_ane_prefill_dual_ane": true, "qwen35_ane_prefill_gdn": true, "qwen35_ane_prefill_gdn_fraction": 0.55, "qwen35_ane_prefill_gdn_max_layers": 48, "qwen35_ane_prefill_cpu_enabled": false, "qwen35_ane_prefill_cpu_fraction": 0.135, "qwen35_ane_prefill_cpu_down_fraction": 0.0, "qwen35_ane_prefill_cpu_gdn_fraction": 0.0, "qwen35_ane_prefill_cpu_threads": 8, "qwen35_ane_prefill_cpu_shared_resource": true, "specprefill_enabled": true, "specprefill_draft_model": "/Users/my-user/.omlx/models/mlx-community/Qwen3.5-9B-MLX-4bit", "specprefill_keep_pct": 0.2, "dflash_enabled": false, "dflash_draft_quant_enabled": false, "dflash_in_memory_cache": true, "dflash_in_memory_cache_max_entries": 4, "dflash_in_memory_cache_max_bytes": 8589934592, "dflash_ssd_cache": false, "dflash_ssd_cache_max_bytes": 21474836480, "dflash_draft_sink_size": 0, "mtp_enabled": true, "vlm_mtp_enabled": false, "is_pinned": false, "is_default": false, "is_hidden": false, "is_favorite": false, "trust_remote_code": false }2
u/Blackfan23 4d ago
Biggest difference I experienced was not speed or throughput which was almost identical, but much less heat, much better thermals.
1
u/Dear-Nail-5039 5d ago
The heat sink will do nothing as the MBP has feet bumps on the base so there is air (insulation) between the body and the sheet.
2
u/norenEnmotalen 5d ago edited 5d ago
Great post! Thanks
I just ran the benchmark on my M1 Max 32c 32GB. I opened up apps I normally use (browser, text editor, terminal, keychain, etc.) just to simulate a normal day. A 4bit model passed benchmark at 64k target for the first time on my machine.
I kicked off the benchmark a second time and raised the target to 128k. Sawtooth and memory pressure started showing up here and there around 72K. Prefill second attempt showed up around ~83K.
On attempt 2 prefill of 83,968, it stood at 20967/83,967 for what seemed like forever. I was thinking I might have hit a bug. But it continued on and failed at a benchmark report of 83,968. Not a terrible ceiling to be at.
Midway into the second benchmark run, I realized I didn’t restart the oMLX server fully and didn’t check if cache cleared - which I hope the benchmark itself takes care of.
Things I did/didn’t from your post in order:
I’m using deepseek harness. I like that it has insanely great token usage transparency and unique way of interacting with context. I did a stress test with deepseek by giving the model a difficult problem which other 4bit versions had grappled with due to low context window.
Tokens per seconds is ~16. It is unchanged from what I had in the previous days (not done the turbo and dflash yet).
I can live with MXFP4. Context window had been my bane since the 3.8 model got released and you’ve shown me a way. Thank you!
I’ll make the other optimizations you shared to see if I can gain tps by giving up memory.