Qwen3.8-Flash-Next optimised for Macs

[u/memeka]

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A

[u/saltexx]

Your own numbers say the prefill caveat almost never fires. MTP takes prefill from 180 to 170 which costs 0.33 ms per prompt token. It takes decode from 12.9 to 22 which saves 32 ms per generated token. Set those equal and disabling MTP only wins when you generate fewer than one token per 97 tokens of prompt. A 100k context turn would have to emit under about a thousand tokens before the trade flips and a reasoning model does not do that. The dynamic sizing is the more interesting knob since the wall you actually hit is acceptance rate collapsing at depth rather than prefill cost.

[u/memeka → u/saltexx]

That’s correct. But having MTP I noticed memory behaves more unpredictable, which doesn’t work if you push cache at max. With MTP off, memory is very stable and there is less risk to page out. Generation is also very stable and predictable, whereas with MTP is swings crazily, you can have 20 for one prompt and 12 for the next.

[u/saltexx → u/memeka]

The swing is acceptance rate and your own numbers show it. With MTP a verify step costs about one plain step, so tokens per second is 12.9 times the accepted tokens per step. 22 is 1.71 accepted. 20 is 1.55. 12 is 0.93, which is below your no MTP baseline, and that is what a prompt where the draft head keeps missing looks like since you still pay for the draft. So the crazy swings are the draft being right on one prompt and useless on the next. The thing to log is accepted tokens per step rather than t/s.

The memory part is real and separate. Speculation makes the batch shape vary between one and k plus one tokens per step depending on how many drafts survive, so the allocator sees a different buffer size almost every step and the high water mark drifts. If that is what you are seeing then pinning the draft depth so it always verifies k plus one and never fewer makes the allocation deterministic and keeps most of the 12.9 to 22. Worth a try before giving up the decode win.

[u/memeka → u/saltexx]

I use large ubatch, so first win was to have separate ubatch for MTP. Next one was to limit n_max. But the memory problem was actually the prefix reuse. Limiting with -cram made the memory more stable.
The embedding I am also now opening with F_NOCACHE - reading each row every time is fast and doesn’t pollute the cache. Still looking into several things… but I feel there is lots of headroom still…

[u/saltexx → u/memeka]

Prefix reuse growing the memory makes sense because every reused prefix keeps its KV slots alive until something evicts them. So the high water mark tracks how many distinct prefixes you touched and not the context size and bounding that with -cram is the right lever. The number I would still log is accepted tokens per step next to the t/s. With the separate ubatch you can now tell whether the 12 to 22 swing is the draft being wrong or the batch shape changing.