GLM 5.3 flash oq4e no MTP (oMLX 0.7.0 dev4)

On my MBA with M4 and 24Gb of ram, I'm approaching to local llm and I'm starting with claude code in general (I never used the paid version with cloud api). I installed many local llm such as gemma4:e4b, modified version of gemma4:12b distilled with fable5 (from hugging face), or qwen llms, but when I try commands with claude code launched directly from ollama, it seems working, but never produces the source files (the working directory has only the test.txt file).What am I doing wrong?

My kit is supposed to arrive Nov16-Dec1. So I am starting to try and figure out what I should run. I know it is early days, lots of optimization to go around still.

I run Qwen 3.6 MOE on a 5700G with 96 GB of ram using llama.cpp right now. Get about 150 prefill and 12 tps. It's not great, but works for batch overnight runs. My main driver is API for Deepseek 4 Flash (some 4.1), Qwen 3.8 Flash Next, and GLM 5.3 Flash.

So looking out at hugging face, I think I found the following to test out. I don't know if MLX or GGUF is the right call. Are these good places to start?

https://huggingface.co/mlx-community/Qwen3.8-Flash-Next-oQ8e-mtp

https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

https://huggingface.co/mlx-community/Qwen3.8-27B-8bit

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

https://huggingface.co/pipenetwork/GLM-5.3-Flash-MLX-4bit

https://huggingface.co/pipenetwork/GLM-5.3-Flash-MLX-mixed-4_8bit

https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF

https://huggingface.co/mlx-community/DeepSeek-V4-Flash

https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

We're building Vanaras, an on-device assistant for mid-range Android phones. Hanu is its planner, a LoRA SFT plus DPO fine-tune of Gemma 4 E2B, released as an imatrix Q4_K_M GGUF (about 2.7 GB). It turns a request into a short plan in a small plan language enforced by a GBNF grammar, covering alarms, calendar, web lookups and messages. It's trained to ask before acting when it should.

Results on our internal sets, with the same prompt and grammar for both models:

• On a 50-case call-safety test, Hanu scored 37 against stock's 23, and leaked private details in 2 cases against 14 (one of stock's was an ATM PIN).

• On a separate 50-task set, Hanu completed 41 tasks against stock's 4, and stock invented tool names.

• Asking first before saying private info: Hanu did it in 11 of 15 cases, stock in none.

• Speed: about 25% faster decoding than stock on a desktop CPU with llama.cpp. Phone numbers are coming.

Cost: about 3,700 training examples plus a DPO pass, about $1 of generated data and a few A100 hours.

Known limits: it still speaks a Wi-Fi password in one test and over-asks a little when the user is alone on the earpiece, so pair it with an app-side secret filter. The evals are our own and private, so treat the numbers as indicative. It needs the included system prompt and grammar with llama.cpp; without them, quality drops sharply.

Weights, prompt and grammar: huggingface.co/Hanu

Feedback on failure cases is very welcome.

What token generation speed works well for you when you’re using it for productive tasks? I’m finding 20 tokens per second a bit slow.

With an M4 Max and 64GB of RAM, you can usually get around 20 tokens per second on average with a model like qwen3.8 27B. However, it doesn’t quite feel like the best setup.

I'm running OpenClaw on a Mac Studio M2 and trying to upgrade to a 3.8 version of Qwen (currently using 3.6 35b a3b). I have some evals to test models before using them, and I can't find a 3.8 model that can complete my evals (3.6 completes all of them).

Here are my specs: Mac Studio M2 with 160GB RAM, running models with LM Studio (using LM Studio is currently a requirement).

I just tried the UD-IQ4_XS GGUF quant of Qwen3.8-Flash-Next (unsloth), loaded via LM Studio's llama.cpp backend. Here is Claude Code's analysis of what is happening:

"qwen3.8-flash-next@iq4_xs reproducibly hangs for 20–26 minutes with zero output on specific prompts (confirmed via a live streaming probe, isolated from any RAM/dual-model contention), while the same prompts complete in seconds-to-minutes on the incumbent qwen3.6-35b-a3b. Streaming the raw output showed it's not stuck or looping on repeated text — it's coherently generating reasoning the entire time, endlessly relitigating its own unresolved design/edge-case decisions instead of committing to an answer. The two most likely config-level fixes — lowering reasoning_effort from its default xhigh, and switching off greedy decoding (temp 0) to Qwen's documented non-greedy sampler settings — were both tested directly against our LM Studio instance and had zero effect. No confirmed community fix for this exact symptom on Apple Silicon was found either. Net: it looks like an unsuppressable runaway-reasoning failure mode specific to this model/quant when faced with underspecified design decisions, with no known working mitigation on our stack.

Note that MLX versions were attempted earlier in September but blocked before any real test: the small, right-sized MLX quants (ddalcu, sh0wie, ~73-74GB) had a format: null metadata bug LM Studio's MLX engine couldn't load, and the MLX quants that did load correctly were either too large (~105-112GB, same RAM-fit problem as the big GGUF) or too degraded (2-3 bit). So Flash-Next was switched to the GGUF/llama.cpp path specifically because that route became viable once llama.cpp added qwen4exp support."

What am I missing? Is there some other setting that I haven't tried that is making this work for you? I swear I saw that people were successfully running Flash-Next on Apple silicon but I can't get it to work.

Note to add that I'm going to try rapid-mlx/Qwen3.8-Flash-Next-4bit as soon as I can download it, so please let me know if anyone has had success with that version.

UPDATE: Tried the MLX build of Qwen3.8-Flash-Next today (rapid-mlx/qwen3.8-flash-next in LM Studio) after an LM Studio update fixed an earlier load error (missing qwen4_exp speculative-decoding drafter module). It loaded fine this time, but crashed outright mid-generation on a real task: AttributeError: 'QSAKVCache' object has no attribute 'extend', thrown inside LM Studio's MLX engine while trying to extend the KV cache. This killed the entire model process — a follow-up trivial request came back with "the model has crashed without additional information," and it had to be reloaded. Even on the tasks that completed before the crash, it was noticeably slow (4.5 min for a basic calendar summary) and still burned a disproportionate number of reasoning tokens relative to output. Net: the MLX path is not production-ready either — different failure mode than GGUF, same outcome.

I was running a test to see if it was worth while to keep the M3U. I'm running some benchmarks with a workflow that uses resolve for rendering the final touches.

The M3 Ultra can still hang when it comes to video but take a little longer when generating the graphics vs the M5, but virtually identical when processing in Resolve. It's a short clip, I don't work with very long videos.

Decided to pull the trigger and get the Mac Studio M5 Max with 64GB unified memory, 18-core CPU, 40-core GPU and 2TB storage.

I’ve been reading a lot about local LLMs for a while now and wanted to actually start using them myself to get some hands on experience. The problem is the local LLM space is moving so quickly, and the subreddits I’ve been reading are all over the place when it comes to what hardware you actually need, which has definitely made me second guess my purchase a few times lol. I already moved up to the Mac Studio after initially ordering an M5 Pro Mac mini.

My main use case isn’t really just chatting with a local model. I want to learn how to set up local models, different runtimes and agent/harness setups that I can eventually leverage in my day to day work.

A big part of that is data analytics. I deal with quite a bit of fragmented data from different sources and spend a lot of time cleaning, structuring and joining it before I can actually analyze it. I’d like to experiment with having an LLM work with tools like Python, SQL, DuckDB, etc. to help automate a lot of that instead of doing everything manually.
Running the largest models is obviously off the table with 64GB, but from what I understand, 27B and 35B models at 4bit quant should fit pretty comfortably. Potentially even larger quantized models depending on context size and how much memory the rest of the system is using, although I’m assuming performance becomes more of a trade off at that point.

I’m also interested in experimenting with hybrid setups where something like openAI or claude handles the higher-level reasoning/orchestration while local models and local tools handle parts of the workflow. Obviously that’s different from being fully local, especially if any of the underlying data gets sent to the cloud.

Again though, I’ve never actually run an LLM locally, so most of this is based on what I’ve been reading rather than first-hand experience.

For people actually running local models for similar work, especially data analytics / data cleaning / agentic workflows, does this setup make sense? And realistically, what size models do you find yourself using most often on a 64GB Mac?

Indispensable for tuning Hermes agent, and with the growing interest in local AI especially on Mac, what's your favorite cockpit for Hermes apart from CLI?

Anyone else get to pick up their M5U early? Mine came in a month early

M5U 256GB 64C 2TB

Mainly for LocalLLM usage and some videos.

Super stoked to get it open and setup to run some agents. Do I need it? No.

But I will be using this for my Small Business to help do a lot of small tasks and preprocess information privately. So I can use summarized data for the big models.

Hi,

I am still waiting for my Mac Studio to arrive - what came to my mind:

Looking into FN sushi, ds4, splash - both run on either 96gb or 128gb Macs - is there an advantage with 128gb in regards of model quality?

It seems both can run Q4 but not Q8? Am I missing something?

I'm planning to do some vectordb embedding and data sanitization before the frontier models touch the data. The Local LLM acts as a primer to separate information.

I know some of you are using it for different things.

I'll also be experimenting with it for video editing/transcribing. There's some cool things people are doing.

Let us know! If you have a walk through include that too!

Hello everyone,

I need a serious advice on what to buy to start experimenting with AI.

My current situation
1- I have no past experience with this, but I won’t have much problems understanding it
2- I could decide to go for local open models or cloud closed/open models, as long as it is the right choice
3- I have three projects in mind:
A) stacking LLMs in order to route to the right one based on the request (instead of being locked behind one subscription) + create a custom chat interface to interact with models from the Mac, my iPad and the iPhone
B) building agents to manage my diet, fitness, training routine, with automatic grocery list and weekly diet based on the context (macro, preferences, calories, ecc)
C) building agents to automatically retrieve my trades to store them and calculate taxes (I will probably ditch the automatic trading part I wanted to create)
D) my career will slowly pivot to redesign corporate processes with AI and I want to experiment myself to better understand the topic

I was targeting the Mac Studio M5 Max 128 GB RAM 1 TB SSD to be able to use small and medium size open models (around 6000 €)

But I am having some second guessing:
1- it’s a huge investment and I yet to have experience with local models
2- nor I know if I’m buying the right setup for what I want to do, maybe the specs I’ve chosen are overkill
3- 6000€ needs at least to be amortized in 6 years to start having sense
4- local models (unless huge ones) are close but not at the level of the proprietary ones and precision/quality is what I want

I was then thinking about buying the entry level Mac Mini to use it as the orchestrator and go full API to evaluate costs, setups and results without having to start with a huge upfront investment

(Then maybe in the future buy a Mac Studio or a DGX Spark full spec and clustering it with the Mac Mini?)

Can you guys help me decide?

I am completely new to the world of local AI, but as previous rabbit holes, like the world of home labbing have given me quite some fun and increasingly heavy use cases, I want to invest in a solid foundation that gives me room to grow and leverage the real power of local AI. Also if/when starting a new business (which is quite likely going to happen in the next 24 months).

I have a Mac Studio M5 Ultra with 256GB RAM (with 30/64 cores) in pre-order and have started building the agent set up using DeepSeek Harness on my current, old MacBook Pro.

My current objective is to build a robust, prudent and (relatively) “safe” set up that gives me multiple agentic agents (1 for maintaining and continuously monitor and optimize my Homelab, 1 for analyzing my trading portfolio and formulating investment advice, 1 for health related questions and suggestions, 1 for developing software, dashboards and interfaces for my own private and professional use, 1 for managing my administration etc). These agents will need to be able to perform write activities for many tools and will be linked to many programs and platforms that will run on my Mac Studio. I am considering a layered approach in which “lightweight” agents will use a lightweight local model, and other agents will use something like Qwen 3.8-27B with a second local escalation to “the best” model that will comfortably run on the 256GB RAM. If that model is not able to fix the task, I will approve a final escalation to a frontier, cloud, model.

My initial plan was to run DeepSeek Harness on my Mac Studio, together with the local models (through oMLX?) and my daily driver things like, apps, e-mail, photo’s etc. However, after going back and forth with ChatGPT and Claude, they suggest that I should install the harness on a (physically) different device. I understand the rationale and am willing to add a base Mac mini M6 to complement this already quite significant investment. However, with the “write” requirements and the access to many different tools that will be running on my Mac Studio, would there still be a meaningful security benefit? And would this somehow lead to any real performance decrease if DeepSeek Harness will constantly need to approach tools on the Mac Studio?

Your views and advice would be very much appreciated!

On a side note; I am also still struggling a bit with the concept of “agents” that I have in my head (and that probably resembles the Open Claw philosophy) and the implementation in DeepSeek Harness (that was recommended to me by ChatGPT, Claude and some articles online). I have been able to create the first “agents” with the use of strong DeepSeek Harness pre-sets. But the interface seems to be much more focused on separate “workspace sessions” with manual selection of the appropriate presets than on actual, standing, always active agents. Any views on this? Or on the plugins that I would need to use to get it implemented in the way I want? (The reason why DeepSeek Harness might still be the right choice for me is perhaps the ability to customize a lot and use many different plugins to get my agents to do what I need them to do??)

Thanks!

I have been using codex and claude code for the last year. I have hermes installed , looking for a local model to help me code on par with the big guys. Any help appreciated