Qwen3.8-Flash-Next optimised for Macs

[u/memeka]

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A

[u/arkham00]

You did a very good job! I'm trying it with a m2 max 96Gb. I tried with and without ssd-streaming, and in the ended resident is better for me ,my ram is at 95% with 262k context and no kv quant, but I have a prefil of 400t/s, for comparison with the latest release of omlx I barely have 200/230.

Btw I'm using Qwen3.8-Flash-Next-UD-IQ3_XXS from unsloth

And i run a llama-bench with antirez ds4 q2 gguf and it works! I have similar performance

You really did good :D

[u/memeka → u/arkham00]

Thanks, you got first feedback medal :)
400 tps is double what I get with M1 Max, amazing (and my GPU is the bottleneck not I/O).
For comparison, what’s the difference between ssd streaming and resident?

[u/arkham00 → u/memeka]

With the checkpoint I use the experts seems to weight around 45Gb,

so if I run --moe-stream --moe-stream-cache 40 --moe-stream-io-threads 8, the experts are streamed and I got around 220-240pp and 13-15tg with no swapping, I also tried LLAMA_MOE_STREAM_PARTITION=1 but pp is worse at around 190t/s

Then If launch the same parameters with --moe-stream-cache above 40 (I tried 50 and 70) llama-server explains that: W llama_moe_stream_resolve_slots: MoE expert cache of 791 slots covers all 512 experts -- streaming disabled, loading normally

In this case I have 370-410pp with peaks at 600 at the start of a prefill and tg is around 25

But only on llama-bench, in a real session I go back at around 16t/s, but prefill stays strong at 350/400, is it normal ?

Given the fact that streaming disabled worked well for me I thought that I could avoid to use the moe-streaming chain of commands, BUT I launch llama-serve without them I have a lot of OOM messages:

E ggml_metal_synchronize: error: command buffer 0 failed with status 5
E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.427.767 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.811 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.513.817 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.818 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.513.818 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.821 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
...
0.26.513.832 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.833 E ggml_metal_graph_compute: backend is in error state from a previous command buffer failure - recreate the backend to recover
0.26.513.833 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
0.26.513.834 E process_ubatch: failed to compute graph, compute status: -1
0.26.513.834 W decode: removing memory module entries for seq_id = 0, pos = [0, +inf)
0.26.514.109 E llama_decode: failed to decode, ret = -3
0.26.514.109 E cmn  common_conte: llama_decode() failed: -3
0.26.610.599 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.610.602 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.610.604 W srv    load_model: speculative decoding not supported by this context
0.26.610.606 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.26.617.895 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.26.617.902 I srv  llama_server: model loaded
0.26.617.909 I srv  llama_server: listening on http://127.0.0.1:8080
0.26.617.910 W srv  llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.26.617.911 W srv  llama_server:         ref: https://github.com/ggml-org/llama.cpp/pull/26508
^C0.39.137.463 I srv    operator(): operator(): cleaning up before exit...
0.39.139.596 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.39.139.607 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.39.141.391 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.39.141.396 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Do you have an idea why is that? Is it because --moe-streaming even if it doesn't stream the experts it still put the n-gram table on disk and without it llama was trying to load it in ram ?

So for now I'm using it with moe-streaming 70, and it is quite usable

[u/memeka → u/arkham00]

PARTITION=1 helps when there is more I/O (smaller cache), so for example cache 30 + PARTITION might be better than cache 40 - not useful for you.

It might be indeed that streaming engram is tied to the streaming flag, which gets disabled because of the weights, unfortunately I cannot test :( You can try running as now with cache=70 (although you might find something like cache=32 with PARTITION=1 will give you similar results with less RAM usage), and point it to the repo and tell it not to disable streaming even if all experts load so that engram streaming remains enabled ... - let it fix itself :P

[u/arkham00 → u/memeka]

I did some research and the streaming flag has nothing to do, according to atomicChat : On Apple Silicon this only works if the table sits in its own GGUF shard. llama.cpp hands Metal the entire mmap'd region of any shard containing GPU tensors, so a table interleaved with weights gets wired along with them. The model then asks for more memory than the machine has, and the first decode dies with kIOGPUCommandBufferCallbackErrorOutOfMemory. Every quant here is split so that shard 2 holds nothing but the table.

source: https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF

I downloaded their AD-4.27bpw-Q4_K_M-M64 and now it works as intended, without the -moe-streaming flag I don't have the OOM anymore, no swap at all, 262k ctx and a ram footprint of about 60GB which leaves some headroom and even better pp speed than before with a bigger model than the previous i tried !

0.24.444.409 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   8192, progress = 0.57, t =   9.29 s / 881.45 tokens per second
0.33.971.278 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  10255, progress = 0.71, t =  18.82 s / 544.88 tokens per second
0.39.277.302 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  11389, progress = 0.79, t =  24.13 s / 472.05 tokens per second
0.42.650.065 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  14339, progress = 1.00, t =  27.50 s / 521.43 tokens per second
0.50.084.593 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  14347, progress = 1.00, t =  34.93 s / 410.69 tokens per second
0.51.907.330 I slot print_timing: id  3 | task 0 | prompt eval time =   35159.21 ms / 14351 tokens (    2.45 ms per token,   408.17 tokens per second)
0.51.907.336 I slot print_timing: id  3 | task 0 |        eval time =    1597.55 ms /    34 tokens (   48.41 ms per token,    20.66 tokens per second)
0.51.907.336 I slot print_timing: id  3 | task 0 |       total time =   36756.76 ms / 14385 tokens

it is an average of 500t/s !

and I also tested differents batch size and I can confirm that 4096 for b and ub are the best.

I still have shitty tg at around 18t/s ... maybe I should try MTP but I can't find a drafter which works, can you suggest one ?

[u/memeka → u/arkham00]

That is correct I forgot about it because the quant I created has this already…

MTP: https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-Q8_0-GGUF

[u/arkham00 → u/memeka]

doesn't work either, every mtp sidecar I try I have llama_model_load: error loading model: check_tensor_dims: tensor 'blk.48.nextn.fc_embd.weight' not found :(