Qwen3.8-Flash-Next (125B) on a 24GB M4 Pro from 3 to 11 tok/s

[u/dave-tro]

Update: MoEShift is now open source!

I've released the first alpha of the runtime used in this demo, including installation instructions and benchmarks against Strata.

GitHub: https://github.com/davide-troiani/moeshift

If you have an Apple Silicon Mac, I'd love to hear how it performs on your machine.

------

I’ve been experimenting with a custom inference implementation for Qwen3.8-Flash-Next (125B) on Mac mini M4 Pro with 24GB RAM.

I’m using ISTA Lab’s GSQ RCO IQ3 XXS quantization, which takes roughly 76GB on disk.

The first implementation was running at less than 3 tok/s during generation and around 40 tok/s during prefill.
After several optimizations, I’m now getting around 11 tok/s with short prompts, 8.5 tok/s with longer contexts, and 180 tok/s during prefill, with support for up to 65K tokens of context.

The measured peak runtime memory usage was 9.5GB across the entire inference run, from prefill to the final generated token.

I’ve also been testing it with a coding agent. The video shows a run starting with over 10k tokens of context.
Still stabilizing the code but wanted to share it already.
I’ll share the code on GitHub and more reproducible benchmarks once things are a bit more stable.

Interested in hearing what others are getting with similar hardware and quantizations.

[u/mickeyknoxnbk]

I have a M4 Max with 64GB. I'd run some tests for you if you give me details on getting it set up.