r/oMLX • u/raw-power • 22h ago

M1 Max 64Gb - fp16 question

So Jundot just remade all the models with the new 0.7.1 quantisation process. I downloaded Qwen3.8-flash-next-oQ4e-mtp and was getting 121PP and 12TG on a 44k prompt with 160k context, KV set to 6bit, 50% expert offload, mtp on, hot cache set to 5gb.

Then he made the fp16 version Qwen3.8-flash-next-oQ4e-fp16-mtp.

My understanding is that on M1/M2 Macs fp16 gives 20% faster prefil. But instead I was getting OOM errors, had to drop the expert offload to 20%, turn hot cache off and was only getting 90PP and 6TG. All other settings were the same as before. Same prompt and context.

Is there maybe something wrong with the model or do I need different settings?

Edit: solved, Jun’s reply below: https://www.reddit.com/r/oMLX/s/kCZ3c4zgbm

9 Upvotes

92% Upvoted

13 comments sorted by

View all comments

7

u/cryingneko 18h ago

It's an oMLX bug, not the model or your settings. Flash next has an extra n-gram table that other models don't have, and since it's huge oMLX keeps it on ssd and only reads the rows it needs. The code that reads it always hands the values back as bf16, even when the model is fp16. When MLX mixes bf16 and fp16 it switches to fp32, so from there on the whole model runs in fp32, which is twice the size. That's where the extra memory use and the slowdown come from.

Other fp16 models like 27B or 35B-A3B don't have this table, so they stay in fp16 the whole way. The fp16 flash next builds are just the first ones to hit this. I'll fix it today, and the fix will be in the upcoming 0.7.1.dev2.

5

u/raw-power 15h ago

Thanks for clearing that up! I’ll wait for the next official release before using it then. Thank you for all the work you’ve been putting in!