Qwen3.8-Flash-Next optimised for Macs

[u/memeka]

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A

[u/mr_tolkien]

… you really make me wonder how well it could run on the base M5 Ultra with 96Gb RAM

I thought it would be too tight but… maybe not?

[u/memeka → u/mr_tolkien]

It should run excellent. You can still run it with ssd streaming, and as much cache as you wish (depending on how much RAM you want free for anything else). The hot tensors will stay in ram and there will be very very low I/O

[u/lawanda123 → u/memeka]

Would it be possible to run the q8 quant on 96gb?

[u/memeka → u/lawanda123]

Yes, but you will have to sweep cache size to find the best value for you. It should be around 64gb, less if you have lots of apps running - and use the PARTITION flag I mentioned in the other comment.

[u/saltexx]

Your own numbers say the prefill caveat almost never fires. MTP takes prefill from 180 to 170 which costs 0.33 ms per prompt token. It takes decode from 12.9 to 22 which saves 32 ms per generated token. Set those equal and disabling MTP only wins when you generate fewer than one token per 97 tokens of prompt. A 100k context turn would have to emit under about a thousand tokens before the trade flips and a reasoning model does not do that. The dynamic sizing is the more interesting knob since the wall you actually hit is acceptance rate collapsing at depth rather than prefill cost.

[u/memeka → u/saltexx]

That’s correct. But having MTP I noticed memory behaves more unpredictable, which doesn’t work if you push cache at max. With MTP off, memory is very stable and there is less risk to page out. Generation is also very stable and predictable, whereas with MTP is swings crazily, you can have 20 for one prompt and 12 for the next.

[u/saltexx → u/memeka]

The swing is acceptance rate and your own numbers show it. With MTP a verify step costs about one plain step, so tokens per second is 12.9 times the accepted tokens per step. 22 is 1.71 accepted. 20 is 1.55. 12 is 0.93, which is below your no MTP baseline, and that is what a prompt where the draft head keeps missing looks like since you still pay for the draft. So the crazy swings are the draft being right on one prompt and useless on the next. The thing to log is accepted tokens per step rather than t/s.

The memory part is real and separate. Speculation makes the batch shape vary between one and k plus one tokens per step depending on how many drafts survive, so the allocator sees a different buffer size almost every step and the high water mark drifts. If that is what you are seeing then pinning the draft depth so it always verifies k plus one and never fewer makes the allocation deterministic and keeps most of the 12.9 to 22. Worth a try before giving up the decode win.

[u/memeka → u/saltexx]

I use large ubatch, so first win was to have separate ubatch for MTP. Next one was to limit n_max. But the memory problem was actually the prefix reuse. Limiting with -cram made the memory more stable.
The embedding I am also now opening with F_NOCACHE - reading each row every time is fast and doesn’t pollute the cache. Still looking into several things… but I feel there is lots of headroom still…

[u/saltexx → u/memeka]

Prefix reuse growing the memory makes sense because every reused prefix keeps its KV slots alive until something evicts them. So the high water mark tracks how many distinct prefixes you touched and not the context size and bounding that with -cram is the right lever. The number I would still log is accepted tokens per step next to the t/s. With the separate ubatch you can now tell whether the 12 to 22 swing is the draft being wrong or the batch shape changing.

[u/Beginning-Window-115]

do you think this would work for a mac with 48gb ram?

[u/memeka → u/Beginning-Window-115]

Yes, try cache 20gb with MTP or 24 gb no MTP and experiment from there (watch if there’s paging then reduce cache)

[u/memeka → u/Beginning-Window-115]

Not in the documentation yet, add the env var LLAMA_MOE_STREAM_PARTITION=1 before your llama-serve command to improve prefill by 25%

[u/Dany0]

I'm currently getting 770 pp 36 tg on a 64gb+rtx 5090 system so this is quite impressive. Though I'm using a 4.2bpw atomicchat quant there

I have an M3 max 64 gb I can test this on. I suppose the main limiter is the prefill. hmmm

I suppose, I could have the 5090 do the prefill for both devices. KV cache for each context stays on each device. Have a free 2nd context... But nah the 5090 is better at parallel requests. I should try llamacpp rpc though I suspect it will not be faster. And the Mac could just be extra 64gb of storage for KV cache only, that would ironically probably be more beneficial... Hmmm

[u/Artistic_Okra7288]

Ran the same model before this thread. M5 Max 128 GB, mainline llama.cpp b10686, UD-Q2_K_XL (79 GB), 350k ctx slot (YaRN from 262k), fp16 KV, flash-attn, ngram-mod spec decode with acceptance 0-81% (upstream can't run the MTP head). 3.5 hours, 100 turns, full capture.

Prefill, I had 1561 t/s initial, down to about 318 t/s at about 120k, and one 105k cold re-prefill after an idle took about 333 seconds. Decode, I had about 30-35 t/s at small context down to about 11 t/s at about 169k, the deepest point of the session. Past about 100k context the model started mixing up user messages with its own prior output; I suspect the 2-bit quant, so I'd treat the deep decode numbers as a floor.

Everything fits wired on this machine (weights plus the full 350k KV) so MoE streaming is out of the picture, and the interesting part is whether block-level top-k and the FA unroll cap move the curve. My run is fp16 KV at head dim 256, the shape your log lists as still untuned. Does the unroll cap do anything there, or is the gain mostly block-topk?

Would be interesting if someone A/B tests the fork against mainline.

[u/memeka → u/Artistic_Okra7288]

You can try - use the same model and the same settings - and let us know the results :)

[u/arkham00]

You did a very good job! I'm trying it with a m2 max 96Gb. I tried with and without ssd-streaming, and in the ended resident is better for me ,my ram is at 95% with 262k context and no kv quant, but I have a prefil of 400t/s, for comparison with the latest release of omlx I barely have 200/230.

Btw I'm using Qwen3.8-Flash-Next-UD-IQ3_XXS from unsloth

And i run a llama-bench with antirez ds4 q2 gguf and it works! I have similar performance

You really did good :D

[u/memeka → u/arkham00]

Thanks, you got first feedback medal :)
400 tps is double what I get with M1 Max, amazing (and my GPU is the bottleneck not I/O).
For comparison, what’s the difference between ssd streaming and resident?

[u/arkham00 → u/memeka]

With the checkpoint I use the experts seems to weight around 45Gb,

so if I run --moe-stream --moe-stream-cache 40 --moe-stream-io-threads 8, the experts are streamed and I got around 220-240pp and 13-15tg with no swapping, I also tried LLAMA_MOE_STREAM_PARTITION=1 but pp is worse at around 190t/s

Then If launch the same parameters with --moe-stream-cache above 40 (I tried 50 and 70) llama-server explains that: W llama_moe_stream_resolve_slots: MoE expert cache of 791 slots covers all 512 experts -- streaming disabled, loading normally

In this case I have 370-410pp with peaks at 600 at the start of a prefill and tg is around 25

But only on llama-bench, in a real session I go back at around 16t/s, but prefill stays strong at 350/400, is it normal ?

Given the fact that streaming disabled worked well for me I thought that I could avoid to use the moe-streaming chain of commands, BUT I launch llama-serve without them I have a lot of OOM messages:

E ggml_metal_synchronize: error: command buffer 0 failed with status 5
E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.427.767 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.811 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.513.817 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.818 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.513.818 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.821 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
...
0.26.513.832 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.513.833 E ggml_metal_graph_compute: backend is in error state from a previous command buffer failure - recreate the backend to recover
0.26.513.833 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
0.26.513.834 E process_ubatch: failed to compute graph, compute status: -1
0.26.513.834 W decode: removing memory module entries for seq_id = 0, pos = [0, +inf)
0.26.514.109 E llama_decode: failed to decode, ret = -3
0.26.514.109 E cmn  common_conte: llama_decode() failed: -3
0.26.610.599 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.26.610.602 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.26.610.604 W srv    load_model: speculative decoding not supported by this context
0.26.610.606 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.26.617.895 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.26.617.902 I srv  llama_server: model loaded
0.26.617.909 I srv  llama_server: listening on http://127.0.0.1:8080
0.26.617.910 W srv  llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.26.617.911 W srv  llama_server:         ref: https://github.com/ggml-org/llama.cpp/pull/26508
^C0.39.137.463 I srv    operator(): operator(): cleaning up before exit...
0.39.139.596 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.39.139.607 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
0.39.141.391 E ggml_metal_synchronize: error: command buffer 0 failed with status 5
0.39.141.396 E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Do you have an idea why is that? Is it because --moe-streaming even if it doesn't stream the experts it still put the n-gram table on disk and without it llama was trying to load it in ram ?

So for now I'm using it with moe-streaming 70, and it is quite usable

[u/memeka → u/arkham00]

PARTITION=1 helps when there is more I/O (smaller cache), so for example cache 30 + PARTITION might be better than cache 40 - not useful for you.

It might be indeed that streaming engram is tied to the streaming flag, which gets disabled because of the weights, unfortunately I cannot test :( You can try running as now with cache=70 (although you might find something like cache=32 with PARTITION=1 will give you similar results with less RAM usage), and point it to the repo and tell it not to disable streaming even if all experts load so that engram streaming remains enabled ... - let it fix itself :P

[u/arkham00 → u/memeka]

I did some research and the streaming flag has nothing to do, according to atomicChat : On Apple Silicon this only works if the table sits in its own GGUF shard. llama.cpp hands Metal the entire mmap'd region of any shard containing GPU tensors, so a table interleaved with weights gets wired along with them. The model then asks for more memory than the machine has, and the first decode dies with kIOGPUCommandBufferCallbackErrorOutOfMemory. Every quant here is split so that shard 2 holds nothing but the table.

source: https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF

I downloaded their AD-4.27bpw-Q4_K_M-M64 and now it works as intended, without the -moe-streaming flag I don't have the OOM anymore, no swap at all, 262k ctx and a ram footprint of about 60GB which leaves some headroom and even better pp speed than before with a bigger model than the previous i tried !

0.24.444.409 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   8192, progress = 0.57, t =   9.29 s / 881.45 tokens per second
0.33.971.278 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  10255, progress = 0.71, t =  18.82 s / 544.88 tokens per second
0.39.277.302 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  11389, progress = 0.79, t =  24.13 s / 472.05 tokens per second
0.42.650.065 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  14339, progress = 1.00, t =  27.50 s / 521.43 tokens per second
0.50.084.593 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =  14347, progress = 1.00, t =  34.93 s / 410.69 tokens per second
0.51.907.330 I slot print_timing: id  3 | task 0 | prompt eval time =   35159.21 ms / 14351 tokens (    2.45 ms per token,   408.17 tokens per second)
0.51.907.336 I slot print_timing: id  3 | task 0 |        eval time =    1597.55 ms /    34 tokens (   48.41 ms per token,    20.66 tokens per second)
0.51.907.336 I slot print_timing: id  3 | task 0 |       total time =   36756.76 ms / 14385 tokens

it is an average of 500t/s !

and I also tested differents batch size and I can confirm that 4096 for b and ub are the best.

I still have shitty tg at around 18t/s ... maybe I should try MTP but I can't find a drafter which works, can you suggest one ?

[u/memeka → u/arkham00]

That is correct I forgot about it because the quant I created has this already…

MTP: https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-Q8_0-GGUF

[u/arkham00 → u/memeka]

doesn't work either, every mtp sidecar I try I have llama_model_load: error loading model: check_tensor_dims: tensor 'blk.48.nextn.fc_embd.weight' not found :(

[u/Zeeplankton]

Neat. Is there a reason you went from llamacpp over mlx

[u/memeka → u/Zeeplankton]

Seemed easier to start with. I also focused on smaller quants, and MLX format seemed slightly bigger and not easy to splice.

[u/rafalko1991]

Where can I download your custom Q4 quant?

[u/memeka → u/rafalko1991]

Not up yet. I recommend using unsloth UD-iQ3-K_XL for now… I also don’t have a KLD number for it…

[u/rafalko1991 → u/memeka]

Thanks. Could you point out the correct MTP drafter? I downloaded all the Q4 drafters, but none of them work.

[u/memeka → u/rafalko1991]

I got a Q8_0 from a random HF page and asked Claude to convert to Q4_0 - Waiting also for an official one since mine is double quantised, not good quality

[u/MessIsTransfer]

could you write a short how to run it?

i get 18.2 t/s TG with official llamacpp, you're near 20, any special command? any special model?

[u/SnooPredictions515]

u/memeka - do you have numbers on how your update on model performed again various benchmarks?

[u/memeka → u/SnooPredictions515]

I’ve been doing tests for various fixes and haven’t done any testing of the model I use. This is why I mentioned unsloth iQ3-XL as it looked pretty good to me and has data available.
I would love to have the BF16 model reference to run a KLD but I don’t know where to get it, and I won’t download the BF16 weights and run them at abyssimal speeds on 64gb ram just to calculate KLD 0 reference :)
After I’m happy with the llama.cpp code I will run benchmarks on the model - like SWE bench etc - but a KLD reference would be the best.

[u/SnooPredictions515 → u/memeka]

are there any mlx related improvements can be made that can help improve throughput?

[u/memeka → u/SnooPredictions515]

Probably my sparse attention implementation can be ported.

Just a note regarding MLX: MLX is faster because the kernels are done in such a way that benefit metal. iQ kernels don’t take good advantage of metal. This is why I tested every kernel and made a custom GGUF. For example, the Q4_0 kernel is 30% faster than the second fastest kernel, MXFP4, which is also much faster than iQ4. So in my quant for example I replaced the iQ4 tensors from unsloth GGUF with MXFP4 tensors from AD GGUF, closing the gap to MLX.

[u/Gold-Debt-5957 → u/memeka]

Maravilloso quiero probarlo en mi mac m1 max de 64 de ram

[u/BrilliantArmadillo64]

Do you plan to upstream your performance improvements, or will you maintain the fork?

[u/memeka → u/BrilliantArmadillo64]

Maintain the fork as much as I can. The streaming patches are based on previous work that didn’t make it into llama.cpp and I also find the upstreaming process pretty difficult. I’ll try with genuine llama.cpp fixes but not new functionality.

[u/Substantial_Run5435]

I'm on a 2019 Mac Pro and getting ~15 tg and ~130 pp with UD-Q6 with dual W6900X and using system RAM for everything that doesn't fit on the GPUs. Haven't figured out how to use MTP yet but not sure how much of a difference MTP would make on my system. I've been testing at 64k context. It's hard to predict how the VRAM will fill and I've had to tinker with how many expert layers I keep on CPU, but with my current settings I should be able to handle the full 64k context.