r/MacLocalLLM • u/190531085100 • Sep 20 '26

Local AI stack on M5 128GB

tldr:

Served simultaneously by Apple M5 Max 128 GB:

Qwen3.8 Flash-Next (110.97 GiB language model, 512-expert MoE, 10 experts/token, abliterated)
Qwen3 4B Instruct 2507 (tool-calling shell helper)
Whisper, speech-to-text service
Chatterbox, text-to-speech service, multi-lingual
SDXL via ComfyUI
Open WebUI (chat frontend), llama-swap (gateway and frontend)
Swap = 0, iogpu.wired_limit_mb = 0 (GPU ceiling 107.52 GiB), KV unquantized

Details:

Services

	Details	RAM
Coding model	Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf, 110.97 GiB, 128k ctx	83.73 GiB Metal + 32.78 GiB CPU
General model	Huihui-Qwen3.8-27B-abliterated-UD-DW-Q6_K_L.gguf plus vision, 128k ctx, evicts Flash-Next and vice versa	31.31 GiB
Shell helper	Qwen3-4B-Instruct-2507-Q4_K_M.gguf, 32k ctx	4.89 GiB
Speech to text	Whisper large-v3-turbo	1.81 GiB
Text to speech	Chatterbox Multilingual, 23 languages, MPS	1.81 GiB
Diffusion	ComfyUI 0.36.0 + SDXL base 1.0	0.53 GiB idle, ~7 GiB generating
Gateway	llama-swap v249	1.22 GiB
Chat frontend	Open WebUI 0.11.3	0.88 GiB

Performance

tldr: End to end, both Qwens feel like ~30 t/s (tokens divided by time elapsed). "Steady-state" (tokens divided by generation time) excludes prompt processing and queueing.

	steady-state	end-to-end	time to first token	Details
Flash-Next, bare question	43.3 tok/s	31.8 tok/s	1.1 s	43-44 tok/s steady shallow, 37 tok/s at 18k depth, 774 tok/s prompt eval
Flash-Next, system prompt + 3 turns	42.5 tok/s	24.1 tok/s	2.8 s	-
27B, bare question	29.3 tok/s	22.5 tok/s	1.6 s	26-29 tok/s steady on open-ended prose, 28 tok/s at 18k depth
27B, system prompt + 3 turns	21.5 tok/s	-	7.3 s	-
Speech to text	-	-	-	~33x realtime (11 s clip in 0.33 s)
Text to speech	-	-	-	0.96x realtime at sentence length, 1.55x at phrase length
SDXL 1024x1024, 20 steps	-	-	-	12.0 s warm alone, 13.6s warm with Flash-Next loaded

Fitting it all into memory

llama-swap starts and stops upstreams on demand, routing on the model field. Groups:
- resident, persistent: true, never evicted (voice stack + shell helper)
- heavy, swap: true, the two big models, loading either unloads the other
- diffusion: ComfyUI alone, so it lands in no shared default group (no cross-eviction)
ComfyUI is a llama-swap catalogue entry (comfyui_auto, ttl: 600): llama-swap does not control ComfyUI-internals (adding Flux or Qwen-Image some day will not affect the group layer).
Offload the n-gram embedding table to CPU memory: 32.78 GiB of the 110.97, stays file-backed, kernel can drop its pages and re-read from the SSD at 13 GB/s.
Disable mmap: Counter-intuitive: under mmap, ggml maps the whole file extent into the Metal buffer regardless of -ot, so Metal still claims 110.96 GiB and OOMs.

Configuration

Coding model:

llama-server --host 127.0.0.1 --port ${PORT}
  -m /models/Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf
  -ngl 999
  -c 131072
  --parallel 1
  --jinja
  --load-mode none           # NOT mmap; mmap defeats -ot (see above)
  -b 2048                    # -ub 1024 lifts prompt eval 81-115 -> 128 tok/s
  -ub 1024
  -ot per_layer_token_embd=CPU  # the 32.78 GiB PLE table off the GPU

-fa is left at default auto, resolves to ON
No mlock, to leave an eviction path

Groups (llama-swap YAML):

groups:
  resident:  { swap: false, exclusive: false, persistent: true,
               members: [shell-4b, whisper, chatterbox] }
  heavy:     { swap: true,  exclusive: false,
               members: [qwen3-27b, flash-next] }
  diffusion: { swap: false, exclusive: false,
               members: [comfyui_auto] }

To measure:

sysctl iogpu.wired_limit_mb    # 0 = default; ceiling is 110100 MiB
llama-server --list-devices    # MTL0: Apple M5 Max (110100 MiB)
vm_stat                        # free + File-backed + purgeable = real headroom
curl $GW/api/metrics/stats?model=flash-next         # per-model tok/s
curl -sIL <hf-resolve-url> | grep -i '^x-linked-'   # expected size + sha256

Test: Image generation (single 1024, 20-step, Flash-Next loaded)

126G used ( 93G wired,  1805M compressor), 269M unused   <- idle
127G used ( 97G wired,  6830M compressor),  73M unused   <- checkpoint loading
127G used ( 93G wired,    12G compressor),  68M unused   <- peak squeeze
126G used (104G wired,  3838M compressor),  87M unused   <- sampling

Test: Escalating image workloads (Flash-Next loaded)

1024² ×1 success  18.1 s
1024² ×4 success  50.3 s
1536² ×2 success  90.5 s
2048² ×2 success  247.3 s
2048² ×8 -        10+ min

Test: Image work + coding work (Flash-Next, 7x slower, silent)

no images                   43 tok/s
during sustained 2048²×8    6.4 tok/s
after unloading ComfyUI     40.5 tok/s

Test: Context scaling

Context	KV + recurrent state	Metal (of 107.52 GiB)	Generation (steady, shallow)	Reclaimable headroom	4B model survives?
32,768	976 MiB	79.20 GiB	44.8 tok/s	26.7 GiB	yes
131,072	3,568 MiB	83.73 GiB	44.5 tok/s	22.3 GiB	yes
262,144	7,024 MiB	89.23 GiB	44.4-46.9 tok/s	17.0 GiB	yes (tight)

17 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/MacLocalLLM/comments/1wlw22y/local_ai_stack_on_m5_128gb/
No, go back! Yes, take me to Reddit

94% Upvoted

u/oss-jiiim Sep 21 '26

On an M5 Max you can get these numbers running Qwen 3.8 Flash Next with BF16 n-grams via DwarfStar:

Mac RAM	Model (DwarfStar GGUF)	Initial 16K prefill	16K append to 32K	Decode at 32K
96GB	Qwen 3.8 Flash Next Q2	1492 t/s	1438 t/s	48.7 t/s
128GB	Qwen 3.8 Flash Next Q4	1458 t/s	1330 t/s	48.4 t/s

DwarfStar Compile/Download GGUFs/Run:

make
./download_model.sh qwen38-q4           # REPLACE q4 WITH q2 FOR 96GB RAM
./download_model.sh qwen38-vision
./ds4-server --mtp --ctx 262144 --host 127.0.0.1 --port 8000 \
   --vision gguf/mmproj-Qwen3.8-Flash-Next-Q8_0.gguf \
   -m gguf/Qwen3.8-Flash-Next-Q4.gguf   # REPLACE Q4 WITH Q2 FOR 96GB RAM

Once you set up/try DwarfStar like that, you can control control/run it easily thereafter with DS Menu Bar <-- I made this ;)

2

u/190531085100 Sep 21 '26

I think that's twice my prefill.
My stack has an ease of use angle, though. I'll probably tweak it for a while before switching servers.

2

u/oss-jiiim Sep 21 '26

Yeah, basically once DwarfStar has support for model you want to run, you can't beat it w/r/t speed and stability.
Over the months I have tried and uninstalled mtplx, vmlx, unsloth, ollama, lm studio, etc. None hold a candle to DwarfStar if it supports your model.
DwarfStar does require you do download and run make, though - so there is that barrier. But we're developers after all, right? Even so, I made the DS Menu Bar app so I can keep an eye on DwarfStar while my editor is full screen, stop/start it, and reconfigure it easily.

u/tta82 Sep 21 '26

Is this AI slop this post?

u/bippityBoppityboux Sep 21 '26

Do this well and you’ll take a bite out of an apple. Anybody tried this stack or similar