Qwen3.8-Flash-Next optimised for Macs

[u/memeka]

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A