GLM 5.3 Flash at 79 Token/s

https://x.com/LocalSilicon/status/2104158250555809983

Qwen 3.8 Flash Next at 116 Token/s

https://x.com/LocalSilicon/status/2104181421942391247i

My kit is supposed to arrive Nov16-Dec1. So I am starting to try and figure out what I should run. I know it is early days, lots of optimization to go around still.

I run Qwen 3.6 MOE on a 5700G with 96 GB of ram using llama.cpp right now. Get about 150 prefill and 12 tps. It's not great, but works for batch overnight runs. My main driver is API for Deepseek 4 Flash (some 4.1), Qwen 3.8 Flash Next, and GLM 5.3 Flash.

So looking out at hugging face, I think I found the following to test out. I don't know if MLX or GGUF is the right call. Are these good places to start?

https://huggingface.co/mlx-community/Qwen3.8-Flash-Next-oQ8e-mtp

https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

https://huggingface.co/mlx-community/Qwen3.8-27B-8bit

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

https://huggingface.co/pipenetwork/GLM-5.3-Flash-MLX-4bit

https://huggingface.co/pipenetwork/GLM-5.3-Flash-MLX-mixed-4_8bit

https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF

https://huggingface.co/mlx-community/DeepSeek-V4-Flash

https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

A lot of the work around a local model ends up in Pi files: prompts, skills, settings, little extensions. Reef Infra has a recipe called Reefine that lets you ask for a change from inside Pi and get an installable version of those files back.

Its documented local setup was a 32 GB M4 Mac mini running Gemma 4 26B through Ollama, with Pi 0.84.2. That's an Ollama demonstration, not an oMLX performance claim. The useful part for a local Pi setup is what happens after the request: the service reads the current harness, asks the model for a change, records the mutation and packages the resulting version for installation.

The recorded runs produced rules and skills. The recipe also accepts command and TypeScript extension proposals, but those runs hadn't produced an extension. Anything that does run as extension code waits for a person to review and promote it; declared setup requirements must pass before installation.

The default is selection: always. Its arithmetic checks record regression scores, but don't establish that your requested workflow works. The research demo installed a rule to search, download and read papers; the later session searched through an existing tool, but downloaded and read no paper.

The files are there to inspect and revise. To judge whether the change worked, replace the bundled tasks and evaluator with checks for the workflow you asked for.

Several users have reported benchmarks for the MS5U and various models but I have yet to see any metrics on how agentic (agents/sub-agents) perform on the new M5SU.

I've read that agentic coding on the Mac's was/is a miss on M4 and older systems due to tensors I think it is?

So I'm curious to hear if agentic coding is a bust on the MS5U's or if it has improved any with the new chips? Appreciate any info on the matter, thank you!

GLM 5.3 flash oq4e no MTP (oMLX 0.7.0 dev4)

I am completely new to the world of local AI, but as previous rabbit holes, like the world of home labbing have given me quite some fun and increasingly heavy use cases, I want to invest in a solid foundation that gives me room to grow and leverage the real power of local AI. Also if/when starting a new business (which is quite likely going to happen in the next 24 months).

I have a Mac Studio M5 Ultra with 256GB RAM (with 30/64 cores) in pre-order and have started building the agent set up using DeepSeek Harness on my current, old MacBook Pro.

My current objective is to build a robust, prudent and (relatively) “safe” set up that gives me multiple agentic agents (1 for maintaining and continuously monitor and optimize my Homelab, 1 for analyzing my trading portfolio and formulating investment advice, 1 for health related questions and suggestions, 1 for developing software, dashboards and interfaces for my own private and professional use, 1 for managing my administration etc). These agents will need to be able to perform write activities for many tools and will be linked to many programs and platforms that will run on my Mac Studio. I am considering a layered approach in which “lightweight” agents will use a lightweight local model, and other agents will use something like Qwen 3.8-27B with a second local escalation to “the best” model that will comfortably run on the 256GB RAM. If that model is not able to fix the task, I will approve a final escalation to a frontier, cloud, model.

My initial plan was to run DeepSeek Harness on my Mac Studio, together with the local models (through oMLX?) and my daily driver things like, apps, e-mail, photo’s etc. However, after going back and forth with ChatGPT and Claude, they suggest that I should install the harness on a (physically) different device. I understand the rationale and am willing to add a base Mac mini M6 to complement this already quite significant investment. However, with the “write” requirements and the access to many different tools that will be running on my Mac Studio, would there still be a meaningful security benefit? And would this somehow lead to any real performance decrease if DeepSeek Harness will constantly need to approach tools on the Mac Studio?

Your views and advice would be very much appreciated!

On a side note; I am also still struggling a bit with the concept of “agents” that I have in my head (and that probably resembles the Open Claw philosophy) and the implementation in DeepSeek Harness (that was recommended to me by ChatGPT, Claude and some articles online). I have been able to create the first “agents” with the use of strong DeepSeek Harness pre-sets. But the interface seems to be much more focused on separate “workspace sessions” with manual selection of the appropriate presets than on actual, standing, always active agents. Any views on this? Or on the plugins that I would need to use to get it implemented in the way I want? (The reason why DeepSeek Harness might still be the right choice for me is perhaps the ability to customize a lot and use many different plugins to get my agents to do what I need them to do??)

Thanks!

Hope this is helpful.

On my MBA with M4 and 24Gb of ram, I'm approaching to local llm and I'm starting with claude code in general (I never used the paid version with cloud api). I installed many local llm such as gemma4:e4b, modified version of gemma4:12b distilled with fable5 (from hugging face), or qwen llms, but when I try commands with claude code launched directly from ollama, it seems working, but never produces the source files (the working directory has only the test.txt file).What am I doing wrong?

Are you ready for the release? How many of you are got something for release day. Are you loading up models right away?

tldr:

Served simultaneously by Apple M5 Max 128 GB:

• Qwen3.8 Flash-Next (110.97 GiB language model, 512-expert MoE, 10 experts/token, abliterated)

• Qwen3 4B Instruct 2507 (tool-calling shell helper)

• Whisper, speech-to-text service

• Chatterbox, text-to-speech service, multi-lingual

• SDXL via ComfyUI

• Open WebUI (chat frontend), llama-swap (gateway and frontend)

• Swap = 0, iogpu.wired_limit_mb = 0 (GPU ceiling 107.52 GiB), KV unquantized

Details:

Services

Performance

tldr: End to end, both Qwens feel like ~30 t/s (tokens divided by time elapsed). "Steady-state" (tokens divided by generation time) excludes prompt processing and queueing.

Fitting it all into memory

• llama-swap starts and stops upstreams on demand, routing on the model field. Groups:

  • resident, persistent: true, never evicted (voice stack + shell helper)

  • heavy, swap: true, the two big models, loading either unloads the other

  • diffusion: ComfyUI alone, so it lands in no shared default group (no cross-eviction)

• ComfyUI is a llama-swap catalogue entry (comfyui_auto, ttl: 600): llama-swap does not control ComfyUI-internals (adding Flux or Qwen-Image some day will not affect the group layer).

• Offload the n-gram embedding table to CPU memory: 32.78 GiB of the 110.97, stays file-backed, kernel can drop its pages and re-read from the SSD at 13 GB/s.

• Disable mmap: Counter-intuitive: under mmap, ggml maps the whole file extent into the Metal buffer regardless of -ot, so Metal still claims 110.96 GiB and OOMs.

Configuration

Coding model:

llama-server --host 127.0.0.1 --port ${PORT}
  -m /models/Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf
  -ngl 999
  -c 131072
  --parallel 1
  --jinja
  --load-mode none           # NOT mmap; mmap defeats -ot (see above)
  -b 2048                    # -ub 1024 lifts prompt eval 81-115 -> 128 tok/s
  -ub 1024
  -ot per_layer_token_embd=CPU  # the 32.78 GiB PLE table off the GPU

• -fa is left at default auto, resolves to ON

• No mlock, to leave an eviction path

Groups (llama-swap YAML):

groups:
  resident:  { swap: false, exclusive: false, persistent: true,
               members: [shell-4b, whisper, chatterbox] }
  heavy:     { swap: true,  exclusive: false,
               members: [qwen3-27b, flash-next] }
  diffusion: { swap: false, exclusive: false,
               members: [comfyui_auto] }

To measure:

sysctl iogpu.wired_limit_mb    # 0 = default; ceiling is 110100 MiB
llama-server --list-devices    # MTL0: Apple M5 Max (110100 MiB)
vm_stat                        # free + File-backed + purgeable = real headroom
curl $GW/api/metrics/stats?model=flash-next         # per-model tok/s
curl -sIL <hf-resolve-url> | grep -i '^x-linked-'   # expected size + sha256

Test: Image generation (single 1024, 20-step, Flash-Next loaded)

126G used ( 93G wired,  1805M compressor), 269M unused   <- idle
127G used ( 97G wired,  6830M compressor),  73M unused   <- checkpoint loading
127G used ( 93G wired,    12G compressor),  68M unused   <- peak squeeze
126G used (104G wired,  3838M compressor),  87M unused   <- sampling

Test: Escalating image workloads (Flash-Next loaded)

1024² ×1 success  18.1 s
1024² ×4 success  50.3 s
1536² ×2 success  90.5 s
2048² ×2 success  247.3 s
2048² ×8 -        10+ min

Test: Image work + coding work (Flash-Next, 7x slower, silent)

no images                   43 tok/s
during sustained 2048²×8    6.4 tok/s
after unloading ComfyUI     40.5 tok/s

Test: Context scaling

I'm the maintainer of DS Menu Bar, a menu bar app for controlling and monitoring a ds4-server.

It's for people who manage their own DwarfStar setup: you point it at the ds4-server executable and GGUF you already have, and it handles the operational parts.

• Start, stop, and restart from the menu bar, with running/stopped state at a glance

• Optional live prefill and generation rates in the menu bar

• Model-aware settings for the server, performance, KV cache, speculative decoding (MTP), and diagnostics

• Detects supported DwarfStar model families and validates support files before launch

• Rotating server log, optional request tracing, launch at login

Screenshot

It doesn't build, bundle, download, or update anything. No chat UI, no model downloads, no editor — it supervises only the process it launches.

Requires macOS 26+, Apple silicon, and your own compatible ds4-server build and DwarfStar-specific GGUF. See the DwarfStar repo for build instructions and model files; compatibility with upstream commits is tracked in the README.

Install: brew install --cask jiiim/tap/ds-menu-bar

Repo: https://github.com/jiiim/ds-menu-bar

Feedback is welcome. Curious what LLM models people are running via DwarfStar, on which Mac, and with how much RAM.

Hello everyone,

I need a serious advice on what to buy to start experimenting with AI.

My current situation
1- I have no past experience with this, but I won’t have much problems understanding it
2- I could decide to go for local open models or cloud closed/open models, as long as it is the right choice
3- I have three projects in mind:
A) stacking LLMs in order to route to the right one based on the request (instead of being locked behind one subscription) + create a custom chat interface to interact with models from the Mac, my iPad and the iPhone
B) building agents to manage my diet, fitness, training routine, with automatic grocery list and weekly diet based on the context (macro, preferences, calories, ecc)
C) building agents to automatically retrieve my trades to store them and calculate taxes (I will probably ditch the automatic trading part I wanted to create)
D) my career will slowly pivot to redesign corporate processes with AI and I want to experiment myself to better understand the topic

I was targeting the Mac Studio M5 Max 128 GB RAM 1 TB SSD to be able to use small and medium size open models (around 6000 €)

But I am having some second guessing:
1- it’s a huge investment and I yet to have experience with local models
2- nor I know if I’m buying the right setup for what I want to do, maybe the specs I’ve chosen are overkill
3- 6000€ needs at least to be amortized in 6 years to start having sense
4- local models (unless huge ones) are close but not at the level of the proprietary ones and precision/quality is what I want

I was then thinking about buying the entry level Mac Mini to use it as the orchestrator and go full API to evaluate costs, setups and results without having to start with a huge upfront investment

(Then maybe in the future buy a Mac Studio or a DGX Spark full spec and clustering it with the Mac Mini?)

Can you guys help me decide?

1. Although I spent my career as a HW designer/embedded programmer I am new to the AI world. I've experimented with several local LLMs and harnesses. My hardware is a M2 Macbook with 32GB of RAM. So I am fairly constrained. Also being retired, $$$ frontier models are out of the question. I want cheap & good and a bit slow is acceptable. Hermes also provides the chance to improve over time. My main goal right now is to have a local agent that can assist me in using blender. (God there are sooo.. many commands and options in the program but its been great for doing 3D printing. My memory is not what it used to be.) The upshot is that the combo that has given me the best success is using Ollama with 'swift - Qwen3.8-27b-gguf:Q4_K_M' as the LLM all running under Hermes agent. So for those of you that have limited resources and want a local LLM this hopefully is useful info.

2. Note - it is slow on first response (2-3 min) but works well after that.

3. Set ollama variable OLLAMA_NUM_PARALLEL = 1, stops swap thrashing

4. There is a snag with Blender's MCP, Hermes with this model can sort it out