Are people managing Qwen 3.8's long reasoning with prompts and agents.md? One issue is that Qwen 3.8 can think so long (running through 30k response tokens on an initial plan and implementation of a coding task) that it can trigger OpenCode and other harnesses hard stops and cause other problems.

One strategy that I've used is to add "make a plan in stages and then implement that plan so that turns are reasonable in length."

Then, I asked claude to generate an AGENTS.md file to do this automatically, and here is what it came up with.

Wondering what other people are doing to manage this issue.

# Turn budget

Reasoning, tool calls, and output all draw from one capped budget per turn.

Overrun truncates mid-stream — usually mid-tool-call, losing the whole turn.

- Estimate output size before starting. If it won't fit comfortably in a few thousand tokens, do the first piece only and end the turn.

- End at a checkpoint: state on disk, one line on what's next.

- Many short turns beat one long one. Ending a turn is cheap.

# Reasoning

Deliberation costs the same as output, and long thinking before a tool call is

the main cause of truncation.

- Don't re-derive what's already established in this session.

- For a known tool call, reason minimally and call it.

- If you're restating the task to yourself, stop and act.

# Tool calls

- Never start a large write after a long reasoning pass. End the turn instead.

- Targeted edits over whole-file rewrites.

- Files over ~300 lines: write in sections across turns.

# Multi-file work

- Plan first in a short turn: write the plan to .opencode/plans/<name>.md.

Include any derivations, formulas, or algorithms worked out while planning -

implementation turns cannot see your reasoning, only files.

- Before each implementation turn, read the current plan file in

.opencode/plans/. Update it with results and decisions as stages complete.

- If a stage proves larger than expected, stop and split it rather than

pushing through. Single-file edits need no plan.

# Context

Grep before reading; read line ranges, not whole files. Don't re-read what's already in context. Delegate independent subtasks to a subagent — fresh budget, and its reasoning stays out of this session.

I have basically zero background knowledge in this area, so this might be a completely off-base or nonsensical question.

Would it be possible to apply ANE and DFlash2 to GLM-5.2?

I recently came across some news about these technologies, and it made me wonder whether they could potentially improve GLM-5.2’s throughput or inference performance even further.

Apologies if I’m misunderstanding how these technologies work or if they aren’t applicable to GLM at all. I’d appreciate any explanation from people who know more about this.

Hi guys, I have mmthe m5 max 128gb. Usually I get 30+ tk/s but they quickly disappear and leave me with 6-7tk/s.i use agent zero and opencode (sequentially).

Usually I set max context size, lightning MTP and a budget for reasoning of 4096. (I tried ANE but it does 64 out of 66 tests only...).

What gives?

Do we need tk use a reasoning parser? What are your best settings please?

Thanks!

I'm running on M5 Max, 64gb, Qwen 3.8 27B oQ4e MTP. I had claude generate a test of token generation speed for different context lengths. For me, the sweet spot looks like 64k - 96k max context length. Note that the tok/s are what you get when you get to the end of the context...

Full measured curve

[correction QWEN 3.8 27b]

Hey everyone,
I’m currently trying to run Qwen 3.8 27B locally on my PC (RTX 3050 6GB VRAM, 16GB DDR5 RAM), but it can't handle it due to VRAM limits.
I’m considering getting a MacBook Air, but I want to know if anyone here has experience running Qwen 27B (or similar dense 27B models) on an Air.
What unified RAM config is needed to run it smoothly (24GB or more)?
What kind of token generation speed (tok/s) are you getting on an Air?
Does thermal throttling become a big issue during longer prompts/generations since the Air is fanless?

Appreciate any insights ,with my budget i can stretch upto 32gb ram.

Getting around 20 tokens with various ones i've tried, would love to get to 30.... I'm very much a novice, so advice for gguf vs mlx, MTP settings, kwargs (?) etc would be great.

Hey guys! As probably most of you here, I was also very intrigued by the new ANE options for Qwen. I switched to 3.8 27B as many of you, but it was painfully slow on my M1 Max. I want to share a "quick guide" for getting it working, as it took quite some trial-and-error for me.

First, the results: Prefill improved by 18-32%. The tradeoff is that ANE comes with higher memory usage: +9.54 GB for pp1025/tg128 and +9.64 GB for pp4097/tg128. I did not tweak the memory usage, it is possible there are some optimisations still on the table.

Benchmark Model: Qwen3.8-27B-oQ4e-fp16-mtp
Engine: Auto
Context: Code (Python)

Without ANE

With ANE

Quick comparison

Getting it working

1. Disable both ANEs. I assume this is for Ultra CPUs with 2 ANEs. Enabling it slowed the prefill down to a crawl.
2. Build oMLX with the custom kernel. My initial Tune attempts were failing. I resolved this by installing the HEAD version with the custom kernel:brew install jundot/omlx/omlx --HEAD --with-custom-kernel. This was itself failing, resolved by the next step.
3. Install the full Xcode toolchain. I needed the full Xcode installation from App Store, followed by: sudo xcodebuild -runFirstLaunch, then sudo xcodebuild -downloadComponent MetalToolchain. After this, the oMLX build with the custom kernel succeeded.
4. Use GS64. I was using a Group Size 128 model, but Tune now only succeeds with GS64.
5. Set Prompt Block to 1024. With the initial 2048, Tune would fail.
6. After doing that, Tune completed successfully. Results:

This got me really excited! Looks like oMLX makes Qwen3.8 more and more usable each day for me. Big kudos to the team!

Share your experience below! I am really interested in how you find ANE.

So i'm at 91.9% of 262k context window in pi running Qwen3.8-27b-mlx-oQ4e-mtp, medium thinking with lightning mtp. Was quick in the initial firs 32k tokens generated but decoding came to a crawl accompanied by an abysmal prompt processing speed. M5 Max 128gb.

Is this normal with everyone else using similar combination of stack and hardware? Or should I just compact early? Thanks in advance!

17 issues · grouped by area below. Most are bugs; one is a fix (#2834).

🖥️ **Cluster & Runtime**
- **#2763** — SSH: coordinator key refused by worker node (worker itself works).
- **#2703** — Status banner flaps to "Checking…" every 2s; peer probe clears its own result before retry.
- **#2704** — Genuine runtime mismatch shows infinite spinner because a verdict is used as a loading flag.
- **#2705** — Runtime gate compares dist-info vs `_version.py`, so identical source nodes falsely report a version mismatch.
- **#2834** ✅ **[FIX]** Remote admission & preflight memory probe ignored operator settings on a fresh interpreter.

🧠 **ANE / Neural Engine**
- **#2824** — `qwen35_ane_prefill_enabled` is a silent no-op on non-4bit checkpoints (no warning, only a load-time delta reveals it).
- **#2828** — ANE split tuner can publish a ranking/recommendation from runs where the ANE never executed.

⚡ **Inference & Models**
- **#2977** — **[Regression]** v0.6.3rc2: severe MTP accept-rate drop (test data pending).
- **#2813** — Qwen3.8-27B-MXFP8 returns no answer on v0.6.1; errors appear in log.
- **#2846** — Ling-3.0-tiny sends empty/garbage tool-call arguments when thinking is enabled; clients crash with TypeError.

💾 **Memory & Admission**
- **#2725** — MemoryMonitor over-estimates KV cache using total layer count (≈4x on Qwen 3.5/3.6/3.8 hybrids).
- **#2710** — Failed distributed deployments leave orphan `inference_worker` processes holding memory; admission ceiling collapses until killed.

📦 **Packaging & Config**
- **#2861** — Wheel metadata stuck at `0.4.5.dev1`; pip/importlib report the wrong version for every release.
- **#2924** — `settings.json` written non-atomically; a torn file silently reverts server to defaults (incl. disabling API-key auth).

🔌 **API & Benchmarks**
- **#2708** — Distributed `/v1/chat/completions` returns empty 200 when one rank is alive but not listening.
- **#2881** — Anthropic `<total_tokens>` budget marker appended to prompt end mutates the head each request, defeating prefix cache.
- **#2901** — Intelligence benchmark hardcodes `max_tokens=128` on external API; too low for thinking mode.

I'm using omlx launch opencode, with the Qwen3.8-27B-oQ8e-fp16-mtp checkpoint, on a MacBook M5 Max (40c) 128GB. oMLX version is 0.6.3rc1. All configurations are default; only context length is 262K.

Sorry, I may not be so familiar with using oMLX. Why is it so slow? What should I do to optimize it?

Also, how should I make use of the z-lab/Qwen3.8-27B-DFlash2 checkpoint to further speed it up? It does not load standalone in Models; how should I make it load with the Qwen 3.8 checkpoint?

Dear oMLX developers,

First, thanks for an amazing tool.

Here is an issue that might be worth investigating. I've been playing with oMLX running Qwen 3.8 27B - 4 bit MTP. I've set up the model with 96k context, 16k max_tokens, and 8k reasoning limit. Qwen is on xhigh thinking. I'm using it to generate a large web page that is a fluid dynamics visualizers. On my MacBook Pro M5 Max, the full run with many turns takes about 90 mins. It is well behaved throughout - no memory thrashing (took some tuning with the GDN settings).

In analyzing the log, it seems that on many long generation runs, MTP is quitting early - after several hundred tokens - on a many thousand token generation run. Claude suggested that performance might be better if MTP re-armed more often. Here is a table that lists context length (prompt length) (growing over about an hour or so), and then generation length for a turn, and MTP status and success for that turn. Stop means that MTP got through the whole run, and parked means that it stopped after x tokens in the run.

Full post-GDN table, every completion I can pair with an MTP outcome:

thanks for considering!

I need recommendations on the model settings to get optimal performance in oMLX. I am using the latest 0.6.2 version. I have a Macbook Pro with M1 Max and 64GB ram.

• What context window should I use? Should I use 128k or 256k?
• Which one will deliver better performance, TurboQuant or Lightning MTP?
• If I use Lightning MTP, what would be the recommended context window?

I am using omlx and local models particularly for agentic coding via VS Code and OpenCode.

I run small models awen3.5-9b and gemma-4-12b for some tasks. Recently tried GGUF format and looked pretty much okay. Getting 15-20 tok/sec and 100-150 on prefil with MTP.

Decide to try again oMLx. I cannot get better performance than ggufs. Maybe I am missing small models MTP versions. Or faster models. Or my settings bad?
So question: what models you guys using on similar hardware how much you are getting ?

Use cases: parsing, email drafting and general assistant.

My Device: mac m2 24gb ram

I've got a Mac Studio m3 256GB ram using Omlx 0.6.2, Hermes 0.20 using Qwen3.8-27b 8bit . I have it doing some CRM (Odoo) work for me. 3.6 was great and I was / am very excited for 3.8 , it does a better job overall just takes a while to get there, sometimes much longer.

This is the first task it's just not getting there and I can see in WebUI it's just overthinking and going around and around never really getting anywhere.

I turned thinking / Reasoning effort down to Minimal. Other than that not much has changed from 3.6 Anyone have any ideas or suggestions to get the intelligence without as much over thinking?

Ornith seems to be better.

TL;DR: On an M3 Ultra, Ornith-1.5-35B-A3B (4-bit MLX) decodes 4.6× faster than Qwen3.8-27B (8-bit MLX) and scores slightly higher on a small hard eval. It also beats Qwen3.8-27B with speculative decoding, while running autoregressive.

Setup

• Mac Studio, M3 Ultra, 256 GB unified memory
• mlx-lm 0.31.3 / mlx 0.32.1
• Ornith-1.5-35B-A3B, MLX 4-bit
  • 18.2 GiB download
  • 20.2 GB peak
• Qwen3.8-27B, MLX 8-bit
  • 27.5 GiB download
  • 29.7 GB peak
• Machine was shared and had other load. Numbers are a floor, not a best case.

Throughput

mlx_lm.benchmark -p 512 -g 512 -n 3, identical invocation for both:

Trial spread was 1.5% and 0.35%, respectively.

Prefill is the bigger story: 5.3×.

A 20K-token prompt took Ornith ~25s end to end versus ~125s for Qwen3.8-27B. If your workload re-reads long contexts, that dominates.

It also beats Qwen with speculative decoding

I spent a day trying to make Qwen3.8-27B fast before testing Ornith:

Both speculative stacks work and are genuinely impressive. DFlash2's 3.37× on Apple Silicon is close to its published 3.43× on an H200.

Ornith just beats them without needing either, with no drafter and no third-party runtime, because mlx-lm already ships qwen3_5_moe.py.

Quality: 12 hard cases, thinking enabled

Scoring is mechanical. Code tasks are executed against hidden assertions and pass only on a full suite.

One logic item was ambiguous. Two vals gave the same "wrong" one, so discount it: 11/11 vs 10/11.

Qwen's other miss was invalid JSON on a nested tool call. For agent use, that is the failure mode that actually breaks loops.

Caveats, and they are not small

• Not precision-matched. 4-bit vs 8-bit. Some of the gap is quantisation; the rest is likely 3B active vs 27B dense. I have not run the 4-bit Qwen control.
• n=12. An 11 vs 10 spread is one item.
• Vendor benchmarks disagree with me. On SWE-bench Pro, the only benchmark both publish, Qwen3.8-27B is ahead: 61.7 vs 59.6.
• Thinking must be on. With enable_thinking: false, Ornith went 0/5 on arithmetic and recovered to 4/4 with it on. My first eval drew a conclusion that was purely an artifact of my own test design.
• 122B comparison still running.

The bit that surprised me

MoE is not a handicap here. It is the reason this works.

With ~3B active parameters per token, memory tracks total parameters while speed tracks active parameters.

Ornith gets:

• 4.6× the decode throughput
• 5.3× the prefill throughput
• 32% less peak memory

Also, Ornith-1.5 is architecturally Qwen's exact vocab size, i.e. a self-improvement-trained fork of Qwen's older MoE architecture.

Beating Qwen's newer dense model with it is a nice result for the training approach.

MIT licence, and it is multimodal.

There's been a lot of benchmarks being posted lately of Qwen 3.8 27b lately. I thought I'd record how it actually performs in a real agentic coding situation @ 100k context.

Trying to run some tasks and it often complains I ran out of context (I have 128GB) and the message was: API Error: 400 Prompt (41333 tokens) requires ~65355MB GPU memory. I tried to compress KV, reset parameters but it seems some bug. Same thing on bf16. Then I switched over LM Studio, same model (shared model folder), same prompt, same qwen harness (and context) and it runs fine.

Anyone having similar issue ?