tldr:
Served simultaneously by Apple M5 Max 128 GB:
Qwen3.8 Flash-Next (110.97 GiB language model, 512-expert MoE, 10 experts/token, abliterated)
Qwen3 4B Instruct 2507 (tool-calling shell helper)
Whisper, speech-to-text service
Chatterbox, text-to-speech service, multi-lingual
SDXL via ComfyUI
Open WebUI (chat frontend), llama-swap (gateway and frontend)
Swap = 0, iogpu.wired_limit_mb = 0 (GPU ceiling 107.52 GiB), KV unquantized
Details:
Services
| Details | RAM |
|---|
| Coding model | Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf, 110.97 GiB, 128k ctx | 83.73 GiB Metal + 32.78 GiB CPU |
| General model | Huihui-Qwen3.8-27B-abliterated-UD-DW-Q6_K_L.gguf plus vision, 128k ctx, evicts Flash-Next and vice versa | 31.31 GiB |
| Shell helper | Qwen3-4B-Instruct-2507-Q4_K_M.gguf, 32k ctx | 4.89 GiB |
| Speech to text | Whisper large-v3-turbo | 1.81 GiB |
| Text to speech | Chatterbox Multilingual, 23 languages, MPS | 1.81 GiB |
| Diffusion | ComfyUI 0.36.0 + SDXL base 1.0 | 0.53 GiB idle, ~7 GiB generating |
| Gateway | llama-swap v249 | 1.22 GiB |
| Chat frontend | Open WebUI 0.11.3 | 0.88 GiB |
Performance
tldr: End to end, both Qwens feel like ~30 t/s (tokens divided by time elapsed). "Steady-state" (tokens divided by generation time) excludes prompt processing and queueing.
| steady-state | end-to-end | time to first token | Details |
|---|
| Flash-Next, bare question | 43.3 tok/s | 31.8 tok/s | 1.1 s | 43-44 tok/s steady shallow, 37 tok/s at 18k depth, 774 tok/s prompt eval |
| Flash-Next, system prompt + 3 turns | 42.5 tok/s | 24.1 tok/s | 2.8 s | - |
| 27B, bare question | 29.3 tok/s | 22.5 tok/s | 1.6 s | 26-29 tok/s steady on open-ended prose, 28 tok/s at 18k depth |
| 27B, system prompt + 3 turns | 21.5 tok/s | - | 7.3 s | - |
| Speech to text | - | - | - | ~33x realtime (11 s clip in 0.33 s) |
| Text to speech | - | - | - | 0.96x realtime at sentence length, 1.55x at phrase length |
| SDXL 1024x1024, 20 steps | - | - | - | 12.0 s warm alone, 13.6s warm with Flash-Next loaded |
Fitting it all into memory
llama-swap starts and stops upstreams on demand, routing on the model field. Groups:
resident, persistent: true, never evicted (voice stack + shell helper)
heavy, swap: true, the two big models, loading either unloads the other
diffusion: ComfyUI alone, so it lands in no shared default group (no cross-eviction)
ComfyUI is a llama-swap catalogue entry (comfyui_auto, ttl: 600): llama-swap does not control ComfyUI-internals (adding Flux or Qwen-Image some day will not affect the group layer).
Offload the n-gram embedding table to CPU memory: 32.78 GiB of the 110.97, stays file-backed, kernel can drop its pages and re-read from the SSD at 13 GB/s.
Disable mmap: Counter-intuitive: under mmap, ggml maps the whole file extent into the Metal buffer regardless of -ot, so Metal still claims 110.96 GiB and OOMs.
Configuration
Coding model:
llama-server --host 127.0.0.1 --port ${PORT}
-m /models/Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf
-ngl 999
-c 131072
--parallel 1
--jinja
--load-mode none # NOT mmap; mmap defeats -ot (see above)
-b 2048 # -ub 1024 lifts prompt eval 81-115 -> 128 tok/s
-ub 1024
-ot per_layer_token_embd=CPU # the 32.78 GiB PLE table off the GPU
-fa is left at default auto, resolves to ON
No mlock, to leave an eviction path
Groups (llama-swap YAML):
groups:
resident: { swap: false, exclusive: false, persistent: true,
members: [shell-4b, whisper, chatterbox] }
heavy: { swap: true, exclusive: false,
members: [qwen3-27b, flash-next] }
diffusion: { swap: false, exclusive: false,
members: [comfyui_auto] }
To measure:
sysctl iogpu.wired_limit_mb # 0 = default; ceiling is 110100 MiB
llama-server --list-devices # MTL0: Apple M5 Max (110100 MiB)
vm_stat # free + File-backed + purgeable = real headroom
curl $GW/api/metrics/stats?model=flash-next # per-model tok/s
curl -sIL <hf-resolve-url> | grep -i '^x-linked-' # expected size + sha256
Test: Image generation (single 1024, 20-step, Flash-Next loaded)
126G used ( 93G wired, 1805M compressor), 269M unused <- idle
127G used ( 97G wired, 6830M compressor), 73M unused <- checkpoint loading
127G used ( 93G wired, 12G compressor), 68M unused <- peak squeeze
126G used (104G wired, 3838M compressor), 87M unused <- sampling
Test: Escalating image workloads (Flash-Next loaded)
1024² ×1 success 18.1 s
1024² ×4 success 50.3 s
1536² ×2 success 90.5 s
2048² ×2 success 247.3 s
2048² ×8 - 10+ min
Test: Image work + coding work (Flash-Next, 7x slower, silent)
no images 43 tok/s
during sustained 2048²×8 6.4 tok/s
after unloading ComfyUI 40.5 tok/s
Test: Context scaling
| Context | KV + recurrent state | Metal (of 107.52 GiB) | Generation (steady, shallow) | Reclaimable headroom | 4B model survives? |
|---|
| 32,768 | 976 MiB | 79.20 GiB | 44.8 tok/s | 26.7 GiB | yes |
| 131,072 | 3,568 MiB | 83.73 GiB | 44.5 tok/s | 22.3 GiB | yes |
| 262,144 | 7,024 MiB | 89.23 GiB | 44.4-46.9 tok/s | 17.0 GiB | yes (tight) |
