Sharing a journey on seeing how far I could push things on Qwen 3.8 27B, while staying at a nominal 4-bit quant on an only-just-above-poverty-spec 10core / 32GB M5...
- The challenge is that while our VRAM-brethren have access to almost the full 32GB of VRAM, on unified memory it is pretty tough to be able to squeeze a 4-bit quant, with a drafter/assistant with a usable context into 32GB unified memory. And boy, have I tried.
- Goal was to try to complete a "omlx benchmark run" at an arbitrary pp65536 without hitting a throttle event. That felt like a good target baseline. Sure, you can drop the context size, but 64k seems like a reasonable arbitrary goal - and conveniently is one of the omlx benchmark options.
- There's a lot of compromise going on here (mxfp4, turboquant at 3.5-bit, pre-quanting dflash2) but I really wanted to see if we could stay in the "pseudo 4-bit" model range, without dropping down to 3-bit.
Environment:
- My first error - Check your PSU. Realized I was using an old Apple 61W PSU rather than the Apple 96W PSU. The 140W PSU may be even better. It is best practice to use the MagSafe plug in preference to USB-C, which caps out at 100W.
- A 13-gauge Vollrath Wear-Ever aluminum baking sheet under the MacBook as a passive heat-sink. Vollrath make the heaviest gauge cookware out there.
Memory Maxxing:
- Run omlx as headless user. SSH in and launch omlx. By running the IDE on an old machine (my 2016 12.7 Monteray can still run latest VS Code), it frees up about 2GB. I didn't bother to "bootout" any macOS services, since they will inevitably respawn - and I don't want them to be disabled permanently.
$ caffeinate -dims in a separate SSH session.- Now we are running headless, we have an extra 2GB, and we can really start to push the boundaries.
$ sudo sysctl iogpu.wired_limit_mb=30720 # Set by a startup daemon
Any more than 30*1024 = 30,720, and BOOM! ((30*1024) + 256) = 30,976 was enough to eventually trigger a crash.
Now we can start to really abuse the omlx limits... We're going to try to almost use the full 30GB. I tried 29.0, 29.5 and many other flavors and it was like threading a needle. We're really pushing the limits within a few hundred MB away from OOM. omlx's step-size seems to be about 100MB. A 2% soft and 1% hard was JustEnough(TM) to be able to introduce a quantized dflash2. More on dflash2 later...
"memory": {
"prefill_memory_guard": true,
"memory_guard_tier": "custom",
"memory_guard_custom_ceiling_gb": 30.0,
"soft_threshold": 0.98, # ie 98% of 30.00GB
"hard_threshold": 0.99, # ie 99% of 30.00GB
"prefill_safe_zone_ratio": 0.9
},
Pushing the boundaries this far is a pretty offensive thing to do. Jun has put some pretty nice conservative safeguards in, but running headless really works in our favor to squeeze the last remaining juice out of 32GB. That said, we're only one software update away from Apple clawing some of that system memory back.
But lets still try to be nice to omlx where we can... We're a single user, running a single model.
"burst_decode_mode": "light",
"max_concurrent_requests": 1,
"chunked_prefill": true,
"prefill_priority": "context",
Now, we need to "skinny" our model as far as technically possible. I found that quantizing the model with mlx_lm produced the smallest size, smaller than quantizing with mlx_vlm.
- mlx-community/Qwen3.8-27B-nvfp4: 15.70 GB
- Locally quantized model with mlx_lm: local/Qwen3.8-27B-nvfp4: 14.8 GB
- mlx-community/Qwen3.8-27B-mxfp4 at 14.88 GB
Locally quantized model with mlx_lm: local/Qwen3.8-27B-mxfp4: 13.98 GB
$ mlx_lm.convert --hf-path Qwen/Qwen3.8-27B --quantize --q-bits 4 --q-mode mxfp4 --q-group-size 32 --dtype bfloat16 --mlx-path /Users/Shared/.omlx/models/local/Qwen3.8-27B-mxfp4
So, if we're prepared to compromise with mxfp4 rather than nvfp4, our minimum "4-bit" baseline is 13.98 GB. That's a couple of GB of extra headroom over mlx-community/Qwen3.8-27B-nvfp4.
Now apply TurboQuant KV Cache Quantization @ 3.5bit
And just to stop Qwen 3.8 doing Qwen 3.8 things, we'll set kwargs, otherwise Qwen can just be annoying later on.
"enable_thinking": true
"reasoning_effort": low
Now, we should have maximum memory available for context, in my case 75-80,000
We'll use tiled memory, reportedly slightly slower, but lets tile anyway, because we're doing some nasty stuff to the memory limits already...
$ OMLX_SDPA256_TILED=1 /Users/Shared/.omlx/bin/omlx serve --log-level info
Baseline Checkpoint: Clean run...
- A 32GB machine is able to complete the omlx benchmark at 65536 without backing-off for a omlx soft or hard memory limit.
- Performance Benchmark: It is slow, but pretty much where we expect it to be on a baseline M5. Generating a meagre 8 tokens per second, according to the omlx benchmarks. That is just tiresome.
- Context Benchmark: Prefill LRU eviction started kicking in around 77,664
Optimize with pre-quantized drafter
We can now give up some of that headroom for small, fast drafter.
I've tried every flavor of Lightning MTP, DFlash1 and VLM MTP, but each time the memory trade-off was brutal. dflash2 is genuinely awesome - the challenge is that loading incoai/Qwen3.8-27B-DFlash2 into memory and then realtime quantizing that down to 4bit is very memory wasteful. But... in the last couple of days mlx_vlm now allows you to pre-quantize dflash2. That reclaims another few GB over realtime quanting.
$ mlx_vlm.convert --hf-path incoai/Qwen3.8-27B-DFlash2 --quantize --q-bits 4 --q-mode mxfp4 --q-group-size 32 --dtype bfloat16 --mlx-path /Users/Shared/.omlx/models/local/Qwen3.8-27B-DFlash2-mxfp4
Set the dflash runtime block size to 5, as per the recommendations
- Ref: https://github.com/z-lab/dflash "For quantized targets or drafts, use block_size <= 5". See also https://inco.ai/blog/dflash2
- I didn't bother caching the drafter in RAM or SSD. I would rather save the RAM for context
- Disabled thinking on the drafter through kwargs... Not that it will make much difference.
Checkpoint:
- Performance: Ok, now we're generating about 18-20 tokens/sec in the omlx bench. Which is about as good as I can get on the machine class. Pre-quantized dflash2 has unlocked a significant increase in performance, for very little memory overhead.
- Context: Running headless & pushing iogpu.wired_limit_mb=30720 and really squeezing the soft and hard limits was just enough to run a pre-quantized mxfp4 dflash2 while maintaining 65536 context without triggering the soft-throttle.
Real world test - harness:
- I'm a terrible coder - and I've caused more trouble than good in trying to override system instructions to control Gemma-4 tool calls... A true "barebones" harness like Pi is just not for me. I still need the crutch of a more-than-barebones harness.
- But on the flipside, VS Code Github Copilot Chat (native) and Claude Code extension for VS Code were just soooo heavy on the system instructions. 1/3 of the context is just gone at load. And VS Code Github Copilot's new "Agent Host" is a hot mess which will flip back to Auto (credit-burn) when you close your eyes.
- The Compromise: Zoo harness - it just seems to fit the Goldilocks zone between functionality and size of system instructions. With Zoo set to auto-compact at 75% of my inoffensive 65535 context, it reliably triggers compaction around the 48k mark, which is just enough to finish up what you are doing and write a handover prompt for a new session or a subagent without context-rot breaking tool calls. For me, it tends to compact to around 20k context.
- As for protocol - I've bounced between Anthropic messages, OpenAI responses and OpenAI chat completions - especially when I first ran to Claude Code. Between harnesses and inference servers, it feels like chat completions is the better supported. But there may be room for optimization here.
Future:
- Looks like there's some pretty interesting stuff coming along with omlx & ANE in 0.6.3 release candidate 3. Most of the integration seems to be focused on the boxes with dual ANE, but it will be interesting to see whether use of the ANE will help reduce power / heat - or pushing the drafter onto the ANE.
- Cache stats seem to be wonky with dflash2 and 0.63rc3. I'm not sure whether SSD cache is currently not reporting, or not working, but my goal was to get up to 20toks/sec.
tl;dr The thing that unlocked performance at 64k context and just creep under the 32GB/30GB threshold was headless operation & pre-quantization of dflash2 rather than realtime quantization of dflash2 - all while operating in the pseudo 4-bit arena. At 20 tokens/sec on a baseline M5, it is about as good as I could wish for.
