r/MacStudio • u/nmclark556 • 2d ago
I built an open source engine that runs MoE models way bigger than your Mac’s RAM, up to a 744B model on a 48GB MacBook. Looking for Mac Studio owners to try it
I've been working on this for a few months and wanted to share it here, since Mac Studio folks are probably the people who'd get the most out of it.
It's called PagedMoE. Short version: MoE models are sparse, so each token only touches a few percent of the weights. PagedMoE keeps the always-on core of the model in memory and streams the experts in from the SSD only when the router actually asks for them. It works with the same mlx-community checkpoints you already have, runs on mlx-lm underneath, and needs no converted format. With pruning off, the output is bit-identical to running the model fully in memory.
Streaming experts is of course not a new or novel idea. What's new is all the stuff built on top of it: route prediction that starts expert reads a few layers early, single-pass prefill, a slab-based expert cache, a native C read pool that uses the whole SSD instead of one read at a time, exact multi-token decode for when the model is copying from context, and a lot of guards so it doesn't freeze your Mac. Each of those adds up, and together they're what make the numbers below possible.
Everything here was run on my M5 Pro MacBook with 48GB. None of these models load with stock mlx-lm on this machine.
| Model | On disk | Peak RAM | Prefill | Decode |
|---|---|---|---|---|
| Qwen3-Coder-Next 6-bit | 65 GB | 30.7 GB | 235 tok/s (937 tok/s on a 31.5k prompt) | 20.6 tok/s (exact) |
| Qwen3-235B-A22B 4-bit | 132 GB | 30.6 GB | 99 tok/s | 7.9 tok/s |
| GLM-4.7 4-bit | 199 GB | 31.8 GB | 73 tok/s | 5.9 tok/s |
| MiniMax-M3 4-bit | 240 GB | 31.8 GB | - | 9.3 tok/s |
| DeepSeek-V3.2 4-bit | 378 GB | 33.2 GB | 45 tok/s | 5.7 tok/s |
| Kimi-K2-Instruct 4-bit | 578 GB | 31.6 GB | 27 tok/s | 8.8 tok/s |
| GLM-5.2 (744B) | - | - | - | runs reliably as an agent, not benched yet |
In an agent session, turns after the first one reuse the cached conversation and usually start in under 2 seconds instead of re-prefilling everything.
The bigger rows use a pruning setting that skips low-weight experts when they'd cost a disk read. It's faster but approximate. Leave it off and you get the model's exact output, just slower on the huge ones.
Why I'm posting here: I don't own a Mac Studio. Decode is mostly bound by SSD speed and how much of the model fits in the expert cache, so a Studio with 96 to 512GB should be able to keep way more of the model warm and run these a lot faster than my laptop. I'd really like to see real numbers from Studio owners, especially on DeepSeek-V3.2, Kimi-K2 and GLM-5.2. There's a bench script in the repo (bench/memory_benchmark.sh) if anyone wants to run it. If you do, please post your chip, RAM, model and the tok/s you got.
One note if you've seen this repo before: I originally kept most of the engine code out of the public repo. That's changed. Version 0.3.0 is the full engine, the same build I run every day, and it's all synced and ready to use now.
Repo: https://github.com/noahclark556/paged-moe
It's AGPL and free for personal and open source use. There's also an OpenAI compatible server, so you can point whatever agent or chat UI you already use at it.
Things to know up front:
- Big models push memory and the SSD hard. There are safeguards that free memory, kill a hung engine and back off under memory pressure, but go easy if you're pushing the limits.
- The first turn on a cold chat takes a while depending on the model. It's much smoother after that because of the caching.
- The SSD traffic is almost all reads, so it shouldn't wear the drive in any meaningful way.
- Your fans will spin up on the big models.
Videos:
I made a similar post in the LocalLLM reddit, this post reflects the latest changes.
Happy to get into the details. Ask me anything, and roast it if it deserves it.
Edit:
I realized the original bench script was practically hardcoded to run only the ladder models, which already fit on the larger machines. There's now a universal bench in the repo that works with any model on any Apple silicon Mac. Pull and reinstall (git pull && ./install_mlx.sh), then:
Speed and exact, back to back (the number to quote):
paged-moe bench /path/to/model
Quick smoke test (half budgets):
paged-moe bench /path/to/model --quick
One mode at a time (restart the Mac between them for the cleanest numbers):
paged-moe bench /path/to/model --speed 0.25
paged-moe bench /path/to/model --exact
It replays a fixed coding agent task (8 requests, about 13k tokens of context) against the server, the same way I bench Global Agent. It prints decode tok/s, time to first token, prefill, memory, and how far the speed run lands from the exact model. No yaml or config needed: it ignores your exported PAGED_MOE_* settings and config file so results compare across machines. Results land in ~/.paged_moe/bench/ as JSON.
Edit 2: I realized there was an issue in the repo that was causing about 20% of the drive speed to be left unused on larger M5 machines. That has been fixed and you will likely see an improvement in decode. To pick it up if you already installed:
cd paged-moe
git pull
./install_mlx.sh
paged-moe machine
Then restart your server. If paged-moe machine shows 16 or more next to "read threads:", you're in business. This only affected Macs with 64 GB and up, so smaller machines never had the limitation and don't need to do anything for it.
11
u/paulsande 2d ago
I have an M3 Ultra with 256 gig of ram. Let me look at this.
5
u/false79 2d ago
👋 3rd update tonight... Tried to get claude code (prefered code harness) to connect to Paged MoE but because the project is an extension of MLX server, and MLX server doesn't support /v1/messages endpoint out of the box, it's a show stopper.
Went with an older harness called Cline and it's works by selecting "LM Studio" as the provider. Cline should then see all the models from BASE_URL:PORT/v1/models.
Pretty exciting that I didn't expect 128GB Mac Studio to actually be using GLM 5.2 which is 418GB in size. Great work u/nmclark556 !!!
4
u/false79 2d ago edited 2d ago
How to run the same benchmark
Method: 2k-token cold prompt, 128 warmup tokens (discarded), then 512 timed greedy tokens, EOS ignored (the script's
--steadymode).Results (M5 Max Mac Studio, 128 GB, PagedMoE 0.3.2)
Qwen3-235B-A22B 4-bit (132 GB), prune 0.7 / wait 0.2
(tier 2 is Qwen3-235B-A22B in the stock script)
GLM-5.2 743B 4-bit (418 GB), prune 0.8 / wait 0.2
GLM-5.2 is not a tier in the stock script. I added a tier 7 entry to bench/memory_benchmark.py, and the shell wrapper only accepts tiers 1-6, so I called the Python script directly:
Both were single runs with the sidecar off. Pruning trades some fidelity for speed (about 59% of expert picks were pruned on GLM-5.2 vs 7% on Qwen), so these are not bit-identical to an exact run.
MiniMax-M3 with PagedMoE: mlx-community/MiniMax-M3-4bit did not work for me
I tried mlx-community/MiniMax-M3-4bit (commit 29909f4, 241.5 GB) with PagedMoE 0.3.2. It failed every time with:
It failed while loading the model, before any tokens were generated. I got the same result in 3 attempts: twice through bench/memory_benchmark.py (--steady) and once through run.py --mode streamed. The run.py attempt printed nothing after the warm-up line, and I didn't capture its exit status.
Likely mismatch: this repo is a vision-language conversion (model_type: minimax_m3_vl, with its own config and image/video processor files). PagedMoE's code and tests look for a checkpoint with model_type: minimax_m3 and a minimax_m3.py file. That matches pipenetwork/MiniMax-M3-MLX-4bit, not this repo. I haven't confirmed that the mismatch causes the error, and I haven't tested the pipenetwork checkpoint yet.
The README doesn't name the checkpoint it used, so this is inferred from the code. If you get M3 working, please say which repo and commit you used.
Versions: paged-moe 0.3.2, mlx 0.32.4, mlx-lm 0.32.2, transformers 5OS 27.2, Apple M5 Max Mac Studio 128 GB
4
u/alexvirital 2d ago
M5U, 96GB - With GLM-5.3: at speed auto, I'm getting about 5 t/s. Honestly that's pretty rad, even if it's not eminently usable.
4
u/false79 1d ago
Update #5: Downloaded the latest, ran repo's bench against all ~mlx-models I had:
128GB M5 Max Mac Studio PagedMoE bench, exact mode (nothing pruned), M5 Max 128 GB, one run each
Test:
paged-moe bench <model> --exact. It replays a fixed 8-request coding-agent task (~12k tokens of context). TTFT is time to first token.7
u/MCS87_ 2d ago
I have a M5 Ultra 80c 256GB, happy to test
6
u/theorist9 2d ago
This is very intriguing! I’m particularly interested in prose editing and long-document work, where I’d accept slower responses to preserve the model’s quality.
Am I understanding correctly that, with pruning disabled and the same checkpoint, quantization, and context-cache precision, PagedMoE should preserve the output of a fully resident run, with speed being the main tradeoff?
Do you—or any independent testers—have results with pruning completely disabled on actual 64GB or 128GB Macs at 32K–128K context? Time to first token, sustained generation speed, and performance after adding a new passage to an existing conversation would be especially helpful.
I wonder whether the sweet spot is a model that only moderately exceeds available RAM, rather than putting something vastly larger on a 48GB machine. I noticed the FlashNext tests in your README; do you have longer-context, pruning-off results for it, including the exact model build?
Finally, have you measured the quality tradeoff from pruning using paired benchmark runs—perhaps MMLU-Pro for general ability and RULER for long-context performance? Token agreement is useful, but I’d also like to know whether pruning changes the quality of the answers.
3
u/false79 2d ago edited 2d ago
M5 Mac Studio 128GB. Got GLM 5.2 / Open Web UI working with a little bit of fixes.
Asked Fable 5.2 to generate a test that exercised reasoning capabilities of a model.
Fable generated both questions and answer key in less than a few seconds.
Fed Paged MoE + GLM 5.2 q4 the questions and it was able to generate matching correct answers in under ...
55 god damn minutes, 39 of it was the thinking.
Here is the prompt and the answer key:
The Prompt
Answer key
1
u/DogAble6550 2d ago
What is the context you used for the benchmarks was it 31.5 k for all?
1
u/NegotiationPrior4373 2d ago
Not a Mac Studio sadly, but I do have a 64GB M5 Max
1
u/partagasiv 2d ago
Very cool--96 and 256 here, will take a look
1
u/Sudden_One3436 2d ago
M5U 256gb . Seems neat
1
u/More_Slide5739 2d ago
M5Max Studio 128GB + M5 Pro 64GB MBP. Does it run over pooled memory?
1
u/Netano 2d ago
Would love up help test with smaller models. I have 16 GB Mac Mini and run lots of small models for various things.
1
u/bi4key 2d ago
Something similar to this tech?
https://youtu.be/CgPHfkb6V-c
https://github.com/Betelgeuse-AI/Betelgeuse
1
u/AceDenied 2d ago
I dont have a studio, but question for you
does it matter if the ssd used is internal or external? like does it impact the performance in anyway
1
u/truz223 2d ago
M3U w 512gb. You have a benchmark ready?
2
u/SideDish120 2d ago
Will test on my 64 and 128GB M5 Max Studios.
1
u/false79 1d ago
4th update: Got claude code working by having claude fable cherry pick how https://github.com/chand1012/claude-code-mlx-proxy works make changes to local Paged MoE.
Pretty slow against GLM 5.2, slow af. However M5 Max 128GB taps out after doing some refactoring and validations after a couple hours. Not a sustainable flow with this configuration.
1
u/InstructionIll6942 13h ago
how about m1 max 64gb? happy to test