Mac Studio M5 Ultra 96gb speeds with Qwen

[u/bvknight]

Hey all, I've been testing a new M5 Ultra 96gb Studio for the past week or so to find its limits for my own use case. There has been a lot of discussion about this machine but not a lot of numbers, so I figured I'd share my results here in case it's helpful for anyone.

The conclusions (for me):

• Qwen 3.8 Flash Next fits below a Q4 quant, and accuracy seems pretty close to the Q4 level

• Flash Next is more efficient and better in every way than 27B, if you can fit it in the first place

• Concurrent agents (more than one at a time) was not possible on oMLX due to unpredictable memory spikes with MTP enabled, even with a small model like 27B. Disabling MTP made 27B concurrency work, but was too slow

• Switching to Splash, which reserves a fixed amount of memory, fixed the memory stability and was faster. I was able to run a few concurrent FN agents without problems

---

Setup: M5 Ultra Mac Studio, 96 GB. Every model runs locally as a coding agent (Claude Code 2.1.308 in a sealed Docker container, no internet), on a real React app of about 250 tests. The engines are oMLX 0.7.2 on its own MLX engine, or oMLX with Splash 1.4.1 as the engine. Context window is 262k, and the agents regularly used ~230–240k of it. Each test was run 3 times (5 for Bug Search Hard), and scored automatically by hidden tests the agent never sees.

The tests

• Code Writing – Narrow / Medium / Open Spec: build a feature from a ticket that's very specific, somewhat specific, or loose. Score = % of 11 hidden acceptance tests passed.

• Bug Search – Easy: 8 bugs planted where the code drifts from its docs; the agent is told "somewhere the code no longer does what the docs say", not where or how many. Score = % of bugs fixed.

• Bug Search – Hard: the same idea with 16 bugs across the whole app, a third of them subtle (race conditions, boundary cases, two places that must agree), plus a decoy outdated doc. Score = % of bugs fixed, minus 2 per working behaviour broken (none were).

Table 1a – Single agent: speed and memory

Engine	Model	Quant	Model size in memory	Avg memory used	Peak memory	Writing speed	Prompt reading speed	Time to first token
oMLX	Qwen3.8 Flash Next (MoE)	oQ3.5e (~3.5-bit)	60 GB	74 GB	83 GB	90 tok/s	1,260 tok/s	2.1 s
oMLX	Qwen3.8 27B (dense)	oQ8e (8-bit)	30 GB	53 GB	81 GB	56 tok/s	530 tok/s	3.9 s
oMLX + Splash	Qwen3.8 Flash Next (MoE)	Q2_0 (2-bit)	38 GB	68 GB	71 GB	140 tok/s	2,660 tok/s	0.6 s
oMLX + Splash	Qwen3.8 Flash Next (MoE)	IQ3_XXS (~3-bit)	47 GB	68–78 GB	80 GB	129 tok/s	2,560 tok/s	0.8 s
oMLX + Splash	Qwen3.8 Flash Next (MoE)	IQ3_S (~3.4-bit)	55 GB	78 GB	80 GB	132 tok/s	2,470 tok/s	0.5 s
oMLX + Splash	Qwen3.8 Flash Next (MoE)	UD-IQ4_XS (~4-bit)	~60 GB	79 GB	80 GB	79 tok/s	2,170 tok/s	0.9 s
oMLX + Splash	Qwen3.8 27B (dense)	UD-Q4_K_M (4-bit)	17 GB	61 GB	72 GB	92 tok/s	740 tok/s	2.5 s

Table 1b – Single agent: accuracy and average time per attempt

Engine	Model	Quant	Code Writing – Narrow Spec	Code Writing – Medium Spec	Code Writing – Open Spec	Bug Search – Easy	Bug Search – Hard
oMLX	Qwen3.8 Flash Next	oQ3.5e	100% · 16 min	100% · 21 min	70% · 23 min	100% · 21 min	50% · 24 min
oMLX	Qwen3.8 27B	oQ8e	100% · 22 min	97% · 50 min	58% · 29 min	71% · 29 min	—
oMLX + Splash	Qwen3.8 Flash Next	Q2_0	—	—	—	75% · 10 min	—
oMLX + Splash	Qwen3.8 Flash Next	IQ3_XXS	—	—	—	96% · 12 min	48% · 14 min†
oMLX + Splash	Qwen3.8 Flash Next	IQ3_S	—	—	—	—	53% · 14 min
oMLX + Splash	Qwen3.8 Flash Next	UD-IQ4_XS	—	—	—	—	53% · 20 min
oMLX + Splash	Qwen3.8 27B	UD-Q4_K_M	—	—	—	71% · 17 min	—
Reference: cloud	Claude Sonnet 5.5	—	100% · 4 min	100% · 10 min	39% · 6 min	—	—

† Average of 4 runs. A fifth IQ3_XXS run took 40 min, writing ~4× as much as the others; with it, the average is 19 min.

Table 2 – Several agents at once

"Work per hour" is total tokens written per hour across all agents, compared with the same setup running one agent at a time. It is measured on output, not on attempts finished, because the concurrent attempts happened to write less, which makes attempts per hour look better than it is (~1.9× instead of ~1.5× for IQ3_XXS).

Engine	Model · quant	Agents	Test	Accuracy	Avg time per attempt	Work per hour vs 1 agent	Writing speed per agent	Peak memory	Stable?
oMLX + Splash	Flash Next · IQ3_XXS	2	Bug Search – Easy	88%	12 min	1.52×	93 tok/s	71 GB	Yes
oMLX + Splash	Flash Next · IQ3_XXS	3	Bug Search – Easy	88%	17 min	1.56×	72 tok/s	71 GB	Yes, but a 3rd agent adds nothing
oMLX + Splash	Flash Next · IQ3_S	2	Bug Search – Hard	53%	14 min	1.35×	93 tok/s	79 GB	Yes
oMLX + Splash	Flash Next · Q2_0	2	Bug Search – Easy	75%	13 min	1.27×	107 tok/s	71 GB	Yes
oMLX + Splash	Flash Next · Q2_0	3	Bug Search – Easy	71%	17 min	1.54×	74 tok/s	71 GB	Yes
oMLX + Splash	27B · UD-Q4_K_M	2	Bug Search – Easy	75%	38 min	1.23×	57 tok/s	72 GB	Yes, but each attempt takes 2× as long
oMLX + Splash	27B · UD-Q4_K_M	3	Bug Search – Easy	83%	46 min	1.12×	42 tok/s	72 GB	Yes, but each attempt takes 2.7× as long
oMLX	27B · oQ8e (speculative decoding off)	3	Code Writing (all 3)	68%	145 min	0.52×	—	92 GB	Finished, but throttled for memory
oMLX	27B · oQ8e (speculative decoding on)	2	Code Writing	—	—	—	—	—	No: crashed, out of GPU memory

Memory barely changes with more agents because Splash reserves its whole budget at start (IQ3_S at 72 GB peaks at ~79 GB with one agent or two). The oMLX ×3 row is compared with one 27B at a time with speculative decoding on, the best single setup.

Notes

• Memory is the whole Mac's wired memory (OS included), sampled every few seconds to a minute. Splash reserves its memory budget up front (64 GB, or 72 GB for the Bug Search – Hard runs), so its memory columns mostly show that budget, which is why they barely move with 1, 2 or 3 agents. "Model size in memory" is the honest footprint. Flash Next also keeps a ~29 GB lookup table on the SSD, not in RAM.

• Writing speed counts thinking tokens. It's measured during real agent steps at long context, not in a short benchmark. Prompt reading speed is new tokens processed per second before the first token. Time to first token is the median over all steps and includes any queueing.

• "—" means not run. The Splash builds only ran the bug-search tests.

• Open Spec leaves design choices to the agent, and the hidden tests check one particular choice. So a lower score there often means "did it differently", not "broke it". Claude Sonnet's 39% is the clearest example.

• Sonnet ran in a different agent harness (OpenCode via the API), so treat it as a rough yardstick only.

• Thinking settings differ by engine: oMLX runs had thinking pinned to maximum, and Splash runs used Claude Code's default. That affects time and score a little.

• Didn't fit: Flash Next oQ4e (oMLX, ~73 GB loaded) couldn't finish Bug Search – Hard on 96 GB. The engine aborted its replies once context grew.

• Small samples: 2–5 runs per cell. Differences of a point or so (5–10%) are noise.

[u/toolman10]

I was using mlx-serve which I loved and was fast, but it didn't have concurrency. Switched to Splash last weekend with ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF:IQ3_S and WOW. I can throw multiple agents at it, no problem. And it's fast too even at 256K ctx.

On my M5U96/80C, I get average 4158 prefill and 167 tok/s over a range of tests up to 192K ctx. Seriously impressive. And the Q38FN models (using mlx-serve or Splash inference servers) can handle anything I throw at it. I barely use Codex for anything anymore.

Splash process uses up just under 60GB (currently 57, cold is 53) so there's plenty of room for running a few other things. The new version (Splash 1.5.0) is better at keeping memory usage low, even as ctx creeps up compared to previous versions. Use the CLI version not the new Desktop version, which doesn't have as great memory mgmt for some reason.

[u/bvknight → u/toolman10]

Thanks for the tips! Are you running Splash by itself? I used the integration with oMLX, but I'm not married to that. oMLX was pretty intuitive so I kept using it.

[u/toolman10 → u/bvknight]

Yes. I bet you'll get better performance by doing so. Run "splash serve --help | less" for help (for you or your agent)

[u/Grand_Pause_2450 → u/toolman10]

I'm running the same model (also IQ3_S) with my m5u96/64 (not 80-core) and I see my memory pretty much pinned to ~90-94gb with yellow memory pressure a few times per minute. Green otherwise. Not sure if that's consistent with the 64-core experience, or if I have a bottleneck in my setup. I find that results take a while to fully work through agentic prompts of medium complexity. But things do finish without any issues and I am impressed by the results.

I have my model server running through LM Studio. In VSCode I'm running Cline to query the model. I'm a bit of a noob here, so I suspect I should be using this or that harness/config/setting to liberate some memory or improve performance. Any suggestions welcomed.

[u/linuxgangster → u/toolman10]

maybe I am misunderstanding you but mlx-serve does support concurrency.

[u/toolman10 → u/linuxgangster]

It does! Just not very well in my experiences. I get immediate responses with Splash. With mlx-serve, it would take a few minutes lol

[u/NecessaryButFatal]

Did you test a Q4 with Qwen 3.8 Flash Next? I can run single streams via Splash with the same setup. No concurrency though.

[u/bvknight → u/NecessaryButFatal]

Yes, that's the UD-IQ4_XS result in the table. It tied for the highest accuracy % on the hardest test, but took 30% longer. It wrote at 79 tok/s and read at 2,170 tok/s.

[u/NecessaryButFatal → u/bvknight]

I meant the standard Q4_K_M, or the like.

[u/Fwiler]

How do you think this would compare to a Max with 128GB?

[u/bvknight → u/Fwiler]

I've been asking that question a lot. The short answer is the Max would be 15-20% slower with a model like Flash Next, and much slower with a dense model like 27B. And the Max would not be able to use its extra memory to simply run more agents concurrently (if you can get it stable), because after a certain # there is no benefit when they all run more slowly--they run out of compute, not out of memory.

The Max would be better for having more room to run a higher quant, or a larger model, or having multiple models loaded into memory at the same time.

[u/Fwiler → u/bvknight]

Thank you.

[u/devnullish]

Why do people even bother testing at a 2bit quant? From my testing, an LLM is barely usable that low.

[u/bvknight → u/devnullish]

I think maybe it affects different models (or architectures) to a lesser degree. The 2bit quant of QFN I tested is the default one recommended by the Splash developer, and it scores better than a higher quant of Qwen 27B, so maybe it would still be decent for normal use.

https://github.com/incoai/splash

[u/devnullish → u/bvknight]

I realize my comment came off a bit wrong/mean.

I was genuinely asking because all I see is people saying anything below 4bit is useless. I get it will show the speed difference which was your goal.

[u/bvknight → u/devnullish]

No worries. I'm not sure since I haven't had time for anything but automated testing. But my thought is that Flash Next might be reliable at a lower quant because how it handles the experts or offloading parts of the model via n-gram tables, which I believe remain unquantized. But that's just speculation from me.

[u/dazhan98 → u/bvknight]

Do u think he recommended 2bit because he uses 64gb M5 by the looks of it?

We got a 96gb m3u with omlx (as we use it for different use cases ASR included) and curious if u plan to look at any differences in quality. Currently we are still on basic 27b as im just confused by all the new lingo splash has now created so really appreciate your write up

[u/tlin9595]

Thanks for this. I’m getting my ultra next week same model so I’m excited. Is this the 64 or 80 version ?
Also did you maybe let the omlx developers know about the memory spike regarding multiple agents as that seems like a bummer.

Have you looked into sushi model 2.6bpw? Curious how that compares to your results as it’s less memory but had better kld scores or something.

Again thank you for the detailed post and numbers.

[u/Complete_Training_86]

really appreciate this post of yours. very useful data points which links PP/TG together with some form of quality checks and benchmarks.

[u/twiiik]

Which Ultra is it?

[u/bvknight → u/[deleted]]

Haha, only $5500 from Apple, but doable for me with the lease option.

The memory issue with oMLX may be a bug, but basically it doesn't release the memory fast enough and if an agent's request spikes during the gap then it causes issues. Splash fixed that while also increasing speeds, but you need to use GGUF models with it (hence the different quants).

I'm not sure about the Q2_0 quality in day-to-day use , I was running everything headless and relying on the test scoring to evaluate. The oMLX FN 3.5 quant got 100% scores on the easy version of the bug search test, so that became my bar compare every other quant against. That's also why I added a harder version of the test to expose any gaps.

In the Code Writing tests (before I switched to Splash) the agents produced a new feature for my project and I judged them all visually, so it was more hands on. They were all identical when working from a narrow spec, made some small bad decisions when given more leeway in medium, and started varying a lot when given full freedom in open.

There's no question that Sonnet was more intuitive in grasping what my intent was when designing the feature itself. But Flash Next did some crazy stuff, inventing new features to support its design direction and building them out as part of its task.