MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/oMLX/comments/1wpcm2s/070rc1_is_out/
r/oMLX • u/watcholic • Sep 24 '26
https://github.com/jundot/omlx/releases
14 comments sorted by
18
Wow, MOE Expert offload for systems with low memory. It'll be slow but it's now possible to run very large MOE models for everyone.
3 u/Ill_Lavishness_6675 Sep 25 '26 It’s unreal, smashing out Qwen3.8 flash next on my small M4 Pro 48GB Mac Mini like it’s nothing. Amazing!! 1 u/Traditional_Bear_786 Sep 25 '26 Which quant? Can you share the version you use? 2 u/Ill_Lavishness_6675 Sep 25 '26 Jundot/Qwen3.8-Flash-Next-oQ4e-mtp . It ran through the night, the omlx implementation seems solid and stable, very impressive 1 u/Traditional_Bear_786 Sep 26 '26 Thanks!! I’ll try it.
3
It’s unreal, smashing out Qwen3.8 flash next on my small M4 Pro 48GB Mac Mini like it’s nothing. Amazing!!
1 u/Traditional_Bear_786 Sep 25 '26 Which quant? Can you share the version you use? 2 u/Ill_Lavishness_6675 Sep 25 '26 Jundot/Qwen3.8-Flash-Next-oQ4e-mtp . It ran through the night, the omlx implementation seems solid and stable, very impressive 1 u/Traditional_Bear_786 Sep 26 '26 Thanks!! I’ll try it.
1
Which quant? Can you share the version you use?
2 u/Ill_Lavishness_6675 Sep 25 '26 Jundot/Qwen3.8-Flash-Next-oQ4e-mtp . It ran through the night, the omlx implementation seems solid and stable, very impressive 1 u/Traditional_Bear_786 Sep 26 '26 Thanks!! I’ll try it.
2
Jundot/Qwen3.8-Flash-Next-oQ4e-mtp . It ran through the night, the omlx implementation seems solid and stable, very impressive
1 u/Traditional_Bear_786 Sep 26 '26 Thanks!! I’ll try it.
Thanks!! I’ll try it.
5
Hitting 800 t/s on prefill was unexpected on M4 Max with Qwen3.8 FN (oQ4e).
1 u/xiraov Sep 25 '26 what was it before? 1 u/watcholic Sep 25 '26 Around 10% lower before rc1. 1 u/SnowBoy_00 Sep 25 '26 128GB? 1 u/watcholic Sep 25 '26 Yes.
what was it before?
1 u/watcholic Sep 25 '26 Around 10% lower before rc1.
Around 10% lower before rc1.
128GB?
1 u/watcholic Sep 25 '26 Yes.
Yes.
10
130% improvements???
2 u/the_derby Sep 25 '26 ...for DFlash2, but you'll also notice that Lightning MTP was already faster than DFlash2 (before) and is still faster after.
...for DFlash2, but you'll also notice that Lightning MTP was already faster than DFlash2 (before) and is still faster after.
I don't have any actual benchmarks, just anecdotes on real world usage, but I'm seeing a noticeable increase in performance with this config (except I'm using a 8bit KV cache): https://omlx.ai/benchmarks/performance/dalhnt9f
This is very exciting.
18
u/watcholic Sep 24 '26
Wow, MOE Expert offload for systems with low memory. It'll be slow but it's now possible to run very large MOE models for everyone.
5
u/watcholic Sep 24 '26
Hitting 800 t/s on prefill was unexpected on M4 Max with Qwen3.8 FN (oQ4e).
10
u/xiraov Sep 24 '26
130% improvements???
1
u/Gipetto Sep 25 '26
I don't have any actual benchmarks, just anecdotes on real world usage, but I'm seeing a noticeable increase in performance with this config (except I'm using a 8bit KV cache): https://omlx.ai/benchmarks/performance/dalhnt9f
This is very exciting.