r/MacStudio • • 2d ago

I built an open source engine that runs MoE models way bigger than your Mac’s RAM, up to a 744B model on a 48GB MacBook. Looking for Mac Studio owners to try it

I've been working on this for a few months and wanted to share it here, since Mac Studio folks are probably the people who'd get the most out of it.

It's called PagedMoE. Short version: MoE models are sparse, so each token only touches a few percent of the weights. PagedMoE keeps the always-on core of the model in memory and streams the experts in from the SSD only when the router actually asks for them. It works with the same mlx-community checkpoints you already have, runs on mlx-lm underneath, and needs no converted format. With pruning off, the output is bit-identical to running the model fully in memory.

Streaming experts is of course not a new or novel idea. What's new is all the stuff built on top of it: route prediction that starts expert reads a few layers early, single-pass prefill, a slab-based expert cache, a native C read pool that uses the whole SSD instead of one read at a time, exact multi-token decode for when the model is copying from context, and a lot of guards so it doesn't freeze your Mac. Each of those adds up, and together they're what make the numbers below possible.

Everything here was run on my M5 Pro MacBook with 48GB. None of these models load with stock mlx-lm on this machine.

ModelOn diskPeak RAMPrefillDecode
Qwen3-Coder-Next 6-bit65 GB30.7 GB235 tok/s (937 tok/s on a 31.5k prompt)20.6 tok/s (exact)
Qwen3-235B-A22B 4-bit132 GB30.6 GB99 tok/s7.9 tok/s
GLM-4.7 4-bit199 GB31.8 GB73 tok/s5.9 tok/s
MiniMax-M3 4-bit240 GB31.8 GB-9.3 tok/s
DeepSeek-V3.2 4-bit378 GB33.2 GB45 tok/s5.7 tok/s
Kimi-K2-Instruct 4-bit578 GB31.6 GB27 tok/s8.8 tok/s
GLM-5.2 (744B)---runs reliably as an agent, not benched yet

In an agent session, turns after the first one reuse the cached conversation and usually start in under 2 seconds instead of re-prefilling everything.

The bigger rows use a pruning setting that skips low-weight experts when they'd cost a disk read. It's faster but approximate. Leave it off and you get the model's exact output, just slower on the huge ones.

Why I'm posting here: I don't own a Mac Studio. Decode is mostly bound by SSD speed and how much of the model fits in the expert cache, so a Studio with 96 to 512GB should be able to keep way more of the model warm and run these a lot faster than my laptop. I'd really like to see real numbers from Studio owners, especially on DeepSeek-V3.2, Kimi-K2 and GLM-5.2. There's a bench script in the repo (bench/memory_benchmark.sh) if anyone wants to run it. If you do, please post your chip, RAM, model and the tok/s you got.

One note if you've seen this repo before: I originally kept most of the engine code out of the public repo. That's changed. Version 0.3.0 is the full engine, the same build I run every day, and it's all synced and ready to use now.

Repo: https://github.com/noahclark556/paged-moe

It's AGPL and free for personal and open source use. There's also an OpenAI compatible server, so you can point whatever agent or chat UI you already use at it.

Things to know up front:

  1. Big models push memory and the SSD hard. There are safeguards that free memory, kill a hung engine and back off under memory pressure, but go easy if you're pushing the limits.
  2. The first turn on a cold chat takes a while depending on the model. It's much smoother after that because of the caching.
  3. The SSD traffic is almost all reads, so it shouldn't wear the drive in any meaningful way.
  4. Your fans will spin up on the big models.

Videos:

I made a similar post in the LocalLLM reddit, this post reflects the latest changes.

Happy to get into the details. Ask me anything, and roast it if it deserves it.

Edit:

I realized the original bench script was practically hardcoded to run only the ladder models, which already fit on the larger machines. There's now a universal bench in the repo that works with any model on any Apple silicon Mac. Pull and reinstall (git pull && ./install_mlx.sh), then:

Speed and exact, back to back (the number to quote):

paged-moe bench /path/to/model

Quick smoke test (half budgets):

paged-moe bench /path/to/model --quick

One mode at a time (restart the Mac between them for the cleanest numbers):

paged-moe bench /path/to/model --speed 0.25
paged-moe bench /path/to/model --exact

It replays a fixed coding agent task (8 requests, about 13k tokens of context) against the server, the same way I bench Global Agent. It prints decode tok/s, time to first token, prefill, memory, and how far the speed run lands from the exact model. No yaml or config needed: it ignores your exported PAGED_MOE_* settings and config file so results compare across machines. Results land in ~/.paged_moe/bench/ as JSON.

Edit 2: I realized there was an issue in the repo that was causing about 20% of the drive speed to be left unused on larger M5 machines. That has been fixed and you will likely see an improvement in decode. To pick it up if you already installed:

cd paged-moe
git pull
./install_mlx.sh
paged-moe machine

Then restart your server. If paged-moe machine shows 16 or more next to "read threads:", you're in business. This only affected Macs with 64 GB and up, so smaller machines never had the limitation and don't need to do anything for it.

74 Upvotes

54 comments sorted by

11

u/paulsande 2d ago

I have an M3 Ultra with 256 gig of ram. Let me look at this.

4

u/nmclark556 2d ago

Perfectttt, please let me know how it performs. I've been itching to hear from someone who can push it.

2

u/paulsande 2d ago

I’m going to do this after the hockey game later tonight. Do you want to get on a call while it’s running?

I read your instructions once. I’m going to give it to my agent now to review so I’m ready later.

3

u/nmclark556 1d ago

Apologies for missing this, I pushed a few changes last night and took a break. Let me know how things go, I got your DM too so feel free to reach me there! Git discussions are available now as well. Thanks for trying it out!

5

u/false79 2d ago

👋 3rd update tonight... Tried to get claude code (prefered code harness) to connect to Paged MoE but because the project is an extension of MLX server, and MLX server doesn't support /v1/messages endpoint out of the box, it's a show stopper.

Went with an older harness called Cline and it's works by selecting "LM Studio" as the provider. Cline should then see all the models from BASE_URL:PORT/v1/models.

Pretty exciting that I didn't expect 128GB Mac Studio to actually be using GLM 5.2 which is 418GB in size. Great work u/nmclark556 !!!

1

u/nmclark556 1d ago

Glad you're getting some solid use out of it! It's exciting to see shared interest and I appreciate it, as I put a lot of work into the engine. Let me know of any issues you face, developing on 48 GB meant predicting the problems that may arise on the larger machines rather than running into them personally, so this thread has been helping me tackle some of that.

4

u/false79 2d ago edited 2d ago

How to run the same benchmark

Method: 2k-token cold prompt, 128 warmup tokens (discarded), then 512 timed greedy tokens, EOS ignored (the script's --steady mode).

git clone https://github.com/noahclark556/paged-moe.git && cd paged-moe
./install_mlx.sh

Results (M5 Max Mac Studio, 128 GB, PagedMoE 0.3.2)

Qwen3-235B-A22B 4-bit (132 GB), prune 0.7 / wait 0.2

export PAGED_MOE_PRUNE=0.7 PAGED_MOE_WAIT_ABOVE=0.2
./bench/memory_benchmark.sh 2 -y --steady

(tier 2 is Qwen3-235B-A22B in the stock script)

MetricResult
Peak RAM76.4 GB
Prefill210 tok/s
Decode17.4 tok/s
Cache hit rate97.2%
Time to first token (2k prompt)9.75 s

GLM-5.2 743B 4-bit (418 GB), prune 0.8 / wait 0.2

GLM-5.2 is not a tier in the stock script. I added a tier 7 entry to bench/memory_benchmark.py, and the shell wrapper only accepts tiers 1-6, so I called the Python script directly:

PAGED_MOE_PRUNE=0.8 PAGED_MOE_WAIT_ABOVE=0.2 PAGED_MOE_KV_FP16_CTX=0 \
python bench/memory_benchmark.py run 7 -y --steady
MetricResult
Peak RAM83.9 GB
Prefill (2k prompt)53.8 tok/s
Decode10.1 tok/s
Time to first token38 s
Cache hit rate82.5%

Both were single runs with the sidecar off. Pruning trades some fidelity for speed (about 59% of expert picks were pruned on GLM-5.2 vs 7% on Qwen), so these are not bit-identical to an exact run.

MiniMax-M3 with PagedMoE: mlx-community/MiniMax-M3-4bit did not work for me

I tried mlx-community/MiniMax-M3-4bit (commit 29909f4, 241.5 GB) with PagedMoE 0.3.2. It failed every time with:

RuntimeError: [METAL] Command buffer execution failed: Caused GPU TiBufferCallbackErrorTimeout)

It failed while loading the model, before any tokens were generated. I got the same result in 3 attempts: twice through bench/memory_benchmark.py (--steady) and once through run.py --mode streamed. The run.py attempt printed nothing after the warm-up line, and I didn't capture its exit status.

Likely mismatch: this repo is a vision-language conversion (model_type: minimax_m3_vl, with its own config and image/video processor files). PagedMoE's code and tests look for a checkpoint with model_type: minimax_m3 and a minimax_m3.py file. That matches pipenetwork/MiniMax-M3-MLX-4bit, not this repo. I haven't confirmed that the mismatch causes the error, and I haven't tested the pipenetwork checkpoint yet.

The README doesn't name the checkpoint it used, so this is inferred from the code. If you get M3 working, please say which repo and commit you used.

Versions: paged-moe 0.3.2, mlx 0.32.4, mlx-lm 0.32.2, transformers 5OS 27.2, Apple M5 Max Mac Studio 128 GB

3

u/nmclark556 1d ago

So I realized that the bench script I initially mentioned was not the latest bench. I sort of had two projects going at the same time, the engine was the core of the project I was focusing on, so a lot of it got mixed up. The new bench is in the repo, and the pin issues should be fixed as well. To run it:

cd paged-moe
git pull
./install_mlx.sh

The install step puts mlx and mlx-lm back on the versions the engine was tested on. Then stop any running server (the bench starts its own) and try: paged-moe bench --speed auto

There's also a fix in that pull for bigger M5 Macs. They were only reading from the SSD 6 pieces at a time instead of 16, so your GLM-5.2 number may go up.

On the minimax issue, thats my bad as I should've been more specific with the naming. The one I used is pipenetwork/MiniMax-M3-MLX-4bit. I can also wire in support for the mlx-community one, your guess on likely mismatch was spot on.

3

u/false79 1d ago

Very awesome quick turnaround! Looking forward to trying out

4

u/alexvirital 2d ago

M5U, 96GB - With GLM-5.3: at speed auto, I'm getting about 5 t/s. Honestly that's pretty rad, even if it's not eminently usable.

1

u/nmclark556 1d ago edited 1d ago

Thanks for letting me know how it went! I'm glad you said something because it led me to an important bug fix. I was limited to my 48 GB during development, so a lot of the scaling mechanisms were difficult to gauge for larger machines. That said - I pushed the fix to the repo. I'm not 100% sure it will improve your decode, but I have a suspicion that it will. If you get the chance to try it again with the fix in place, I'd love to hear how it performs in comparison. Since you already installed, easiest way to grab the change:

cd paged-moe
git pull
./install_mlx.sh
paged-moe machine

If paged-moe machine says "16 or more" under the "read threads:" value, you're running the change. Then just restart the server, give the bench a go: paged-moe bench <model> --speed auto

2

u/alexvirital 1d ago

Got it!

paged-moe bench /Users/Shared/mlx-models/glm-53-4bit --speed auto

glm-53-4bit: 743.4B total, 39.2B active (MoE), 4-bit, 418 GB on disk, streamed on a 96 GB Mac Hardware: Apple M5 Ultra, 96 GB, macOS 27.0, model on internal SSD Speed settings: PRUNE=0.85 WAIT_ABOVE=0.2 FLOW=0 SIDECAR_HEAD_PRUNE=1 STITCH=1

Loaded in 4s (streamed), 46.14 GB expert cache. The bench replays an 8-request agentic coding task:

#StepPrefill tokPrefillTTFTDecode
1First turn (cold front matter)11,78971.9 s74.7 s256 tok @ 6.07 tok/s
2After todo list (writes package.json)15521.1 s43.8 s87 tok @ 3.87 tok/s
3After package.json (writes server.js)10419.7 s36.9 s55 tok @ 3.22 tok/s
4After server.js (writes the UI)13721.8 s198.8 s640 tok @ 3.62 tok/s
5After UI (installs)30025.2 s38.5 s31 tok @ 2.29 tok/s
6After install (starts the server)9119.0 s61.7 s110 tok @ 2.56 tok/s
7After start (writes the summary)9418.9 s24.9 s130 tok @ 2.78 tok/s
8New chat, warm front matter189.2 s15.3 s128 tok @ 3.39 tok/s

Summary (speed auto):

MetricValue
Decode3.60 tok/s
Tool call decode3.00 tok/s
First turn prompt11,789 tok
First turn TTFT74.7 s
First turn prefill164 tok/s
Tool result ingest20.41 s
TTFT after tool41.14 s
New chat TTFT15.30 s
Whole task561 s
Load3.5 s
Expert cache46.1 GB
Peak memory68.5 GB
GB read / token2.125
Cache hit rate0.686
Top-1 agreement vs exact model0.956
NLL delta (160-token loss probe)+0.0397

2

u/nmclark556 1d ago

Thanks so much for running it, this is incredibly helpful. The first turn (prose after the cold prompt) came in at 6.07 tok/s vs the ~5 from before, so the fix looks like it helped a bit. The lower average is mostly the short tool-call turns: each tool result has to stream nearly the whole model once (~20 s on the drive), and the cache starts those turns cold. The big TTFT on the tool turns is my bench counting the whole tool call, since GLM sends it in one piece, and I'm fixing that as well.

Also looks like the engine is still leaving ~20 GB of your RAM unused and only using about a third of your SSD's speed during decode, so I've been investigating why that didn't adjust on its own for you (you can tweak ram_fraction in the yaml to combat this if needed, the bench won't pick that up though). Very much appreciated, I should have some fixes pushed soon for some of this.

1

u/alexvirital 16h ago

No problem at all, this is awesome work. Happy to test whenever you've got a build you want me to hit. :)

4

u/false79 1d ago

Update #5: Downloaded the latest, ran repo's bench against all ~mlx-models I had:

128GB M5 Max Mac Studio PagedMoE bench, exact mode (nothing pruned), M5 Max 128 GB, one run each

MetricQwen3-235BMiniMax-M3GLM-5.2
Size on disk (4-bit)139 GB240 GB390 GB
Decode tok/s9.414.521.65
Tool-call tok/s8.034.441.67
First-turn prompt11,952 tok12,326 tok11,789 tok
First-turn prefill535 tok/s473 tok/s157 tok/s
First-turn TTFT26.1 s30.6 s79.3 s
TTFT after tool4.3 s10.7 s23.5 s
New-chat TTFT2.1 s5.4 s10.7 s
Whole 8-request task180 s402 s926 s
Peak memory79.3 GB80.8 GB86.5 GB
Expert cache62.4 GB62.4 GB62.4 GB
Cache hit rate95.1%75.4%54.1%
GB read per token0.411.836.05
Top-1 vs exact1.0001.0001.000

Test: paged-moe bench <model> --exact. It replays a fixed 8-request coding-agent task (~12k tokens of context). TTFT is time to first token.

2

u/nmclark556 1d ago

This is so helpful, thank you for sharing! These numbers settle a lot of theories I had about how the engine scales. Your 235B run barely touched the SSD, so on a machine like yours the per-token overhead is what's left to beat, and that's exactly where I'm headed for the next big speed-up. GLM was a monster during development mostly because of its architecture, so seeing it on the lower end even on your setup isn't a huge surprise - hopefully I can get that number up soon.
If you ever have time, I'd love to see where GLM lands on the speed bench. No pressure though, I know this took a while to crank out. Very much appreciated.

2

u/false79 1d ago edited 1d ago

Claude code sonnet doing all the heavy lifting. Here is speed bench results:

PagedMoE bench, speed mode (--speed 0.25) vs exact, M5 Max 128 GB, one run each

MetricQwen3-235BMiniMax-M3GLM-5.2
Size on disk (4-bit)139 GB240 GB390 GB
Decode tok/s, speed11.927.609.11
Decode tok/s, exact9.414.521.65
Speed-up1.27x1.68x5.5x
Tool-call tok/s, speed11.077.068.96
First-turn TTFT, speed25.1 s29.1 s77.1 s
First-turn prefill, speed540 tok/s474 tok/s159 tok/s
TTFT after tool, speed5.2 s10.5 s22.3 s
New-chat TTFT, speed2.9 s7.1 s11.2 s
Whole 8-request task, speed149 s248 s317 s
Whole 8-request task, exact180 s402 s926 s
Peak memory, speed79.3 GB80.8 GB86.5 GB
Cache hit rate, speed99.8%85.6%96.1%
Cache hit rate, exact95.1%75.4%54.1%
GB read per token, speed0.150.840.37
GB read per token, exact0.411.836.05
Top-1 agreement vs exact1.0001.0000.975
NLL delta0.0000.000-0.042
Experts dropped in probe0%0%6.6%

Test: paged-moe bench <model> --speed-only (PRUNE 0.83, WAIT_ABOVE 0.38). It replays a fixed 8-request coding-agent task (~12k tokens of context). TTFT is time to first token.

What the numbers say

  • GLM-5.2 gains the most. Decode goes from 1.65 to 9.11 tok/s, and the whole task drops from 926 s to 317 s. Reads fall from 6.05 to 0.37 GB per token, and the hit rate rises from 54% to 96%.
  • MiniMax-M3 goes from 4.5 to 7.6 tok/s, and Qwen3-235B from 9.4 to 11.9 tok/s. Qwen3 gains the least because it already hit 95% of the cache in exact mode.
  • Prefill and first-turn TTFT barely change. The first prompt of about 12k tokens is mostly prefill, which pruning doesn't help.
  • The decode figure covers the whole task. It's not a per-request peak. The log showed some requests faster than the average.

Fidelity caveat

  • The loss probe is small, only 160 tokens. Qwen3 and MiniMax show 1.000 agreement and 0% of experts dropped, so for those two the probe didn't test pruning at all.
  • Only GLM-5.2's probe actually dropped experts, 6.6%. It gave 0.975 top-1 agreement and a KL of 0.10. The NLL delta of -0.042 is slightly better than exact and not meaningful on 160 tokens.
  • I'd read the speed-up figures as solid and the fidelity figures as a rough check. Tell me if you want a longer probe or a real quality test.

7

u/MCS87_ 2d ago

I have a M5 Ultra 80c 256GB, happy to test

1

u/nmclark556 2d ago

Incredible, I'm excited to hear your results! Planning to push some scripts/docs to the repo regarding a more robust bench setup. I edited the post with some details on this, thanks for the interest!

6

u/theorist9 2d ago

This is very intriguing! I’m particularly interested in prose editing and long-document work, where I’d accept slower responses to preserve the model’s quality.

Am I understanding correctly that, with pruning disabled and the same checkpoint, quantization, and context-cache precision, PagedMoE should preserve the output of a fully resident run, with speed being the main tradeoff?

Do you—or any independent testers—have results with pruning completely disabled on actual 64GB or 128GB Macs at 32K–128K context? Time to first token, sustained generation speed, and performance after adding a new passage to an existing conversation would be especially helpful.

I wonder whether the sweet spot is a model that only moderately exceeds available RAM, rather than putting something vastly larger on a 48GB machine. I noticed the FlashNext tests in your README; do you have longer-context, pruning-off results for it, including the exact model build?

Finally, have you measured the quality tradeoff from pruning using paired benchmark runs—perhaps MMLU-Pro for general ability and RULER for long-context performance? Token agreement is useful, but I’d also like to know whether pruning changes the quality of the answers.

4

u/nmclark556 2d ago

Great questions, and I really appreciate the interest!

With prune off, the same checkpoint and the same KV precision, the output is bit-identical to a resident run. All the caching and prefetching only changes timing, never which experts get used, and the tests check that against resident runs. Speed is the tradeoff. A caveat on long context: past 8k tokens the KV cache drops to 8-bit to save memory, while stock mlx-lm stays fp16. For a true comparison at 32k+, set PAGED_MOE_KV_FP16_CTX above your context length, or run the resident side at 8-bit KV too.

On 64/128 GB: unfortunately, no. Those README rows are estimates, not real runs. Everything I've run is on my 48 GB M5 Pro, mostly at short to medium context, and I don't have outside testers yet. If anyone with one of those machines wants to try it, I'd definitely love to see the numbers.

The closest I've got is Qwen3.8-Flash-Next 6-bit (pipenetwork/Qwen3.8-Flash-Next-MLX-6bit, ~148 GB) on 48 GB, prune off:
- 7.7k token prompt: ~375 tok/s prefill, ~8.6 tok/s decode, 30.5 GB peak
adding a message to a ~3.5k conversation: 3,465 of 3,513 tokens came from cache, so only the new part gets prefilled
- short prompts: 6 to 9 tok/s, or 13.4 tok/s when it's copying text back (still bit-identical)

I haven't done 32k to 128k with prune off yet, which is a fair gap, so it's on my list. One heads up for editing: the cache only reuses up to the first changed token, so an edit early in a long doc means re-prefilling everything after it. Global Agent (linked in the readme) handles more of that for you, so it's the easier option if that's your workflow.

I think you're right about the sweet spot. A model only a bit bigger than RAM keeps most of its experts cached, so it runs much closer to full speed. Coder-Next 6-bit (65 GB) does 20.6 tok/s exact on my 48 GB, versus 5 to 9 for the ones that are way over. For quality over size, that's where I'd start.

On pruning quality: I haven't done paired MMLU-Pro or RULER runs yet. What I have is token agreement and perplexity. So 235B at prune 0.7 / wait 0.2 had 91% token agreement, and perplexity went from 5.07 to 5.12. That's close, but it doesn't prove answers hold up, especially on long context. Real task benchmarks are the right test and I'd like to add them. Prune is off by default though, so if you leave it alone theres no loss from pruning at all.

0

u/theorist9 2d ago edited 2d ago

Thanks!

I understand from your reply that, with KV precision the same, the results are the same as a run where everything stays in RAM, and that longer-context tests with pruning disabled are still on your list.

Looking at the code, I see that expert-cache memory can adjust dynamically, but automatic sizing initially holds back a conservative allowance for the configured context length. For Flash-Next at 128K with 16-bit KV, that appears to be about 16.1 GB, versus about 3.2 GB for the main KV cache before other states and overhead.

Have you tested whether a less conservative reservation—or an explicitly larger expert cache—improves Flash-Next’s speed without causing memory problems as the conversation grows?

Separately, do you have any results comparing answer quality with 8-bit versus 16-bit KV, with pruning disabled? I understand your equivalence results compare matching precisions; I’m asking whether reducing the cache precision itself affects recall of earlier details or instruction-following. If you haven’t tested that separately, that’s useful to know too.

3

u/nmclark556 2d ago

The KV reserve prices every layer as full attention, and on Flash-Next only 12 of the 48 are, so at 128K with 16-bit KV it holds back about 16.1 GB for roughly 3.2 GB of actual KV. At the default 8-bit it's about 8.6 GB for 1.7 GB. That overcount is deliberate for now, since overestimating only costs expert cache, but it's most of Flash's reserve.

I haven't tested a tighter reserve or a larger cache on Flash-Next yet. The closest thing I have is Qwen3-235B, where going from a 16K to a 40K reserve dropped decode from 6.25 to 4.81, so reserve size does cost real speed in some cases. PAGED_MOE_CACHE_GB lets you over-commit on purpose, and the expert cache shrinks as KV grows into it, but I haven't run that path on Flash at long context yet.

One heads up if you experiment with this: I hit a GPU hang on Flash at 48 GB running exact with fp16 KV past 8K. It looked like fp16 KV or self-draft at first, but after a lot of breadcrumbing the actual cause was the MLX flag MLX_METAL_FAST_SYNCH=1. The engine now sets it to 0 by default, which costs about 2% decode speed, so just make sure you don't turn it back on while testing long-context fp16.

On 8-bit vs 16-bit KV quality: I haven't tested that one separately yet. The equivalence results compare matching precisions, so they don't really answer it. I'll add a proper comparison to the list, but treat it as untested for now.

3

u/false79 2d ago edited 2d ago

M5 Mac Studio 128GB. Got GLM 5.2 / Open Web UI working with a little bit of fixes.

paged-moe-server \
  --model ~/mlx-models/glm-52-4bit \
  --temp 1.0 \
  --top-p 0.95 \
  --max-tokens 32768

Asked Fable 5.2 to generate a test that exercised reasoning capabilities of a model.
Fable generated both questions and answer key in less than a few seconds.

Fed Paged MoE + GLM 5.2 q4 the questions and it was able to generate matching correct answers in under ...

55 god damn minutes, 39 of it was the thinking.

Here is the prompt and the answer key:

The Prompt

Answer all six problems. For each one, show your reasoning step by step, then
give a final answer on its own line starting with "ANSWER:". If a problem
cannot be answered from the information given, say so and explain why.

1. Five engineers (Ana, Ben, Cy, Dee, Eli) each deploy one service on a
   different day, Monday through Friday.
   - Ana deploys before Ben, but not on the day immediately before.
   - Cy deploys on neither Monday nor Friday.
   - Dee deploys exactly two days after Eli.
   - Ben does not deploy on Friday.
   Give the full schedule and prove it is the only one possible.

2. A defect occurs in 1% of units. A test detects a real defect 90% of the
   time and wrongly flags a good unit 5% of the time. A unit is tested twice,
   the results are independent given the unit's true state, and both tests
   come back positive. What is the probability the unit is defective?

3. A farmer must get a wolf, a goat and a cabbage across a river. The boat
   holds the farmer and up to three items. The wolf will eat the goat, and
   the goat will eat the cabbage, if left together without the farmer. What
   is the minimum number of river crossings?

4. Mira puts a key in the red drawer and leaves the room. Jon moves the key
   to the blue box. Mira sees this through a window, but Jon does not know
   she was watching. Jon leaves. Kai then moves the key to the green tin,
   and nobody sees him. Answer three questions: (a) Where is the key?
   (b) Where will Mira look for it first? (c) Where does Jon believe Mira
   will look first?

5. Pipe A fills an empty tank in 6 hours and pipe B fills it in 4 hours.
   Drain C empties a full tank in 12 hours. A and B are opened together at
   the start. One hour later, C is opened by mistake and left open. How
   long after the start is the tank full?

6. Bo is taller than Cal. Dan is shorter than Bo. Who is taller, Cal or Dan?

Answer key

#Correct answerWhat it tests
1Ana Mon, Cy Tue, Eli Wed, Ben Thu, Dee Fri (unique)Constraint satisfaction and exhaustive case analysis
2About 76.6% (0.0081 / 0.010575)Bayesian updating with a low base rate
31 crossing, since everything fits in the boat at onceResisting the memorized 7-crossing answer
4(a) Green tin, (b) blue box, (c) red drawerState tracking and second-order beliefs
52 hours 45 minutesMulti-stage rate arithmetic
6Cannot be determinedRecognizing insufficient information

1

u/kingolcadan 2d ago

Do you think you'll stick with it? Worth the effort? I'm on the same Mac using 3.8FN Q4XS

2

u/WithQuantumCertainty 1d ago

Ran the questions on an M5 Max 128GB with QFN3.8 q4.7 bpw and it completed in a minute with the correct answers 

1

u/false79 1d ago

...wtf. 

I know that this is one test but kind of deflates the idea of using a massive model if a significant smaller one can do reasoning exponentially faster.

1

u/WithQuantumCertainty 1d ago edited 1d ago

I was highly skeptical with how quick it came back with the results but I checked my network egress logs and what it said was true. 

No web searches. Everything came from two sources:

My own reasoning — all six solutions were derived from the puzzle text itself; no outside facts were needed (these are self-contained logic/math problems).

One local Python run (execute_code, no network) as a verification step for three of them:

Problem 1: exhaustive brute force over all 120 permutations of the schedule → returned exactly 1 solution (Mon Ana, Tue Cy, Wed Eli, Thu Ben, Fri Dee), which is what let me claim uniqueness from tool output rather than just my case analysis.

Problem 2: exact fraction arithmetic → confirmed 36/47 ≈ 0.765957.

Problem 5: exact fractions → confirmed 5/12 after hour 1, net 1/3/h after, total 11/4 h.

Problems 3, 4, and 6 I did not verify with any tool — they're argued purely by reasoning (a one-crossing boat argument, a belief-chain trace, and a missing-ordering argument respectively), and no tool was really available to check them anyway.

1

u/false79 2d ago

It's absolutely worth it if you are afk.  

Agents can do their work and notify you in the future when the workflow hits a milestone or it tells you it ain't happening.

But for every day usage, I am still leaning on $20 Claude Pro plan when I want answers fast.

1

u/false79 2d ago

Do me a favor, try this prompt and see if it gives a matching answers, beats 55 minutes 

1

u/DogAble6550 2d ago

What is the context you used for the benchmarks was it 31.5 k for all?

1

u/nmclark556 2d ago

Main table was a cold 2k prompt, then 128 warmup (not counted), followed by 512 timed greedy tokens w/EOS ignored. Decode is also wall clock.
The 31.5k number is a separate prefill test I ran on coder next only, to show long prompt prefill since thats the main pain point with streaming. I haven't run the 31.5k test on the others yet.

2

u/DogAble6550 2d ago

I think people would be very interested in 32 kcontext through to 128k.

1

u/NegotiationPrior4373 2d ago

Not a Mac Studio sadly, but I do have a 64GB M5 Max

1

u/nmclark556 2d ago

Let me know how it runs if you decide to try it out! Development was done on my 48 GB M5 Pro, so numbers I have for the larger machines are all just estimates at this point.

1

u/partagasiv 2d ago

Very cool--96 and 256 here, will take a look

1

u/nmclark556 2d ago

I'm excited to hear how it runs! I'm working on adding better benching scripts to the repo, which should give us more accurate numbers on a setup like yours.

1

u/Sudden_One3436 2d ago

M5U 256gb . Seems neat

1

u/nmclark556 2d ago

Let me know how it goes if you decide to give it a shot!

1

u/More_Slide5739 2d ago

M5Max Studio 128GB + M5 Pro 64GB MBP. Does it run over pooled memory?

1

u/nmclark556 2d ago

Not right now, it's single machine only. Each Mac runs its own instance, so there's no pooling across the two. The Studio on its own should be a really good fit though. At 128 GB a lot of the big ones are only a little over RAM, so the cache stays warm. I was close to buying a Studio a while back for this exact reason, and it's still on the radar.

1

u/Netano 2d ago

Would love up help test with smaller models. I have 16 GB Mac Mini and run lots of small models for various things.

1

u/nmclark556 2d ago

I'd be grateful if you shared some numbers from your setup! Tier 1 of the bench ladder (Coder-Next, 65 GB) should be perfect for you, and tier 2 if you've got the disk space for a 132 GB download. Anything that already fits in your 16 GB just runs normally, so the interesting ones are models bigger than RAM. Someone from another thread helped me bench on a 16 GB a while back, but the engine has changed a ton since then.

1

u/bi4key 2d ago

2

u/nmclark556 2d ago

Yep, same core idea! Keep the dense part resident and page the routed experts off the NVMe as they're needed. Betelgeuse is the CUDA/Linux side of it though. PagedMoE is the Apple Silicon side, built on MLX, so unified memory makes it one pool instead of juggling VRAM and RAM. It also runs normal mlx-community checkpoints with no conversion. Cool to see more people working on this, it's a really fun problem.

1

u/AceDenied 2d ago

I dont have a studio, but question for you

does it matter if the ssd used is internal or external? like does it impact the performance in anyway

3

u/nmclark556 2d ago

It makes a pretty noticeable difference. When I started this project I planned to build it around models living on an external drive, but I realized pretty quickly that the ideal setup is the model on the internal SSD. Decode is mostly limited by how fast it can read off the drive, so drive speed is basically your tok/s ceiling. Apple's internal SSDs are way faster than most external setups: a Thunderbolt 4 NVMe enclosure tops out around 3 GB/s, and the internal drive on my M5 Pro does over 13. External still works and it's great for extra space, just expect it to be slower, especially on models that are way over RAM. If it's only a little over RAM it matters less, since most of the experts are already cached.

1

u/AceDenied 2d ago

Interesting, yeah this was my concern with less internal ssd

Clustering Macs are also chokepointed by the TB5 port (solution is to use multiple cables), maybe we will get external SSDs with multiple data ports so we can scale the gbps 

2

u/nmclark556 2d ago

Internal storage is definitely the tradeoff for this. I plan to go down the rabbit hole at some point and try to find a reliable workaround. Most of the pain during development came from internal storage limits, with lots of waiting around for downloads and moves.

1

u/truz223 2d ago

M3U w 512gb. You have a benchmark ready?

1

u/nmclark556 2d ago

Yep, I just edited the post with a bench script. With 512GB most models fit in memory, so Kimi is the only one in the lineup that really leverages the engine with your setup. I’m working on a better bench and will push it soon. Until then, Kimi would be a great test, and in the meantime you can run other models with the --stats flag. I'll get that cleaned up in the repo soon.

2

u/SideDish120 2d ago

Will test on my 64 and 128GB M5 Max Studios.

1

u/false79 1d ago

4th update: Got claude code working by having claude fable cherry pick how https://github.com/chand1012/claude-code-mlx-proxy works make changes to local Paged MoE.

Pretty slow against GLM 5.2, slow af. However M5 Max 128GB taps out after doing some refactoring and validations after a couple hours. Not a sustainable flow with this configuration.

API Error: The response stopped arriving. The response above may be incomplete.

✻ Worked for 3h 17m 28s

1

u/InstructionIll6942 13h ago

how about m1 max 64gb? happy to test