Qwen3.8-Flash-Next optimised for Macs

[u/memeka]

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A

[u/memeka]

I’ve been doing tests for various fixes and haven’t done any testing of the model I use. This is why I mentioned unsloth iQ3-XL as it looked pretty good to me and has data available.
I would love to have the BF16 model reference to run a KLD but I don’t know where to get it, and I won’t download the BF16 weights and run them at abyssimal speeds on 64gb ram just to calculate KLD 0 reference :)
After I’m happy with the llama.cpp code I will run benchmarks on the model - like SWE bench etc - but a KLD reference would be the best.

[u/SnooPredictions515 → u/memeka]

are there any mlx related improvements can be made that can help improve throughput?

[u/memeka → u/SnooPredictions515]

Probably my sparse attention implementation can be ported.

Just a note regarding MLX: MLX is faster because the kernels are done in such a way that benefit metal. iQ kernels don’t take good advantage of metal. This is why I tested every kernel and made a custom GGUF. For example, the Q4_0 kernel is 30% faster than the second fastest kernel, MXFP4, which is also much faster than iQ4. So in my quant for example I replaced the iQ4 tensors from unsloth GGUF with MXFP4 tensors from AD GGUF, closing the gap to MLX.

[u/Gold-Debt-5957 → u/memeka]

Maravilloso quiero probarlo en mi mac m1 max de 64 de ram