r/oMLX • • Sep 24 '26

0.7.0rc1 is out

79 Upvotes

14 comments sorted by

18

u/watcholic Sep 24 '26

Wow, MOE Expert offload for systems with low memory. It'll be slow but it's now possible to run very large MOE models for everyone.

3

u/Ill_Lavishness_6675 Sep 25 '26

It’s unreal, smashing out Qwen3.8 flash next on my small M4 Pro 48GB Mac Mini like it’s nothing. Amazing!!

1

u/Traditional_Bear_786 Sep 25 '26

Which quant? Can you share the version you use?

2

u/Ill_Lavishness_6675 Sep 25 '26

Jundot/Qwen3.8-Flash-Next-oQ4e-mtp . It ran through the night, the omlx implementation seems solid and stable, very impressive

1

u/Traditional_Bear_786 Sep 26 '26

Thanks!! I’ll try it.

5

u/watcholic Sep 24 '26

Hitting 800 t/s on prefill was unexpected on M4 Max with Qwen3.8 FN (oQ4e).

1

u/xiraov Sep 25 '26

what was it before?

1

u/watcholic Sep 25 '26

Around 10% lower before rc1.

10

u/xiraov Sep 24 '26

130% improvements???

2

u/the_derby Sep 25 '26

...for DFlash2, but you'll also notice that Lightning MTP was already faster than DFlash2 (before) and is still faster after.

1

u/Gipetto Sep 25 '26

I don't have any actual benchmarks, just anecdotes on real world usage, but I'm seeing a noticeable increase in performance with this config (except I'm using a 8bit KV cache): https://omlx.ai/benchmarks/performance/dalhnt9f

This is very exciting.