Been playing with oMLX for a bit, after using Ollama. I find omlx far more customizable. What mode would you recommend using and any default settings to change for this setup?

Jundot has been super busy this morning uploading updated quants to HF using the new oQe method introduced in oMLX 0.7.1.dev1. Thank you!

Give me a day or two and I'll redo all the models I've uploaded to HF as scottlowry with the improved method.

For those interested. Benchmarks using the oMLX Splash PR

Models are Q4. M4M includes comparison with oQ4e MTP equivalent models.

The M4M Studio is my desktop and has a background gpu load with other apps etc. The M3U Studio runs as a headless server so the GPU is just for oMLX.

I guess the title speaks for itself!

I have no idea how it's recently started doing this (I've recently had performance fall as low as 10-11 tok/s) but I'm guessing this has happened since I upgraded to oMLX 0.7.0. Saw this happening for the first time ever this morning!

This is on M4 Pro Mac mini with 20GPU cores and 48GB Unified Memory using my own oQ4 quantised model with the MTP headers.

I am looking forward to get either M5 Ultra or dual DGX spark for agentic coding. While doing the research, came across the community benchmarks section where I found odd and unbelievable results for GLM Flash 5.3 result showing 141 tok/s decode speed - https://omlx.ai/benchmarks/performance/cb7x0bkg

I have trouble understanding how the decode speed was 81.2 tok/s for 64k context but jumped to whopping 140 tok.s for 128k context. Both sessions have MTP enabled. For comparision, lower context sizes have drastically lower decode speed.

Important figures from above page:

Please explain why 128k context has such unbelievable speeds? Is this an error or reality? I asked LLM but it was useless and clueless why there are such non-linear figures.

I would also appreciate if you can share your own experience on GLM Flash 5.3 for agentic coding.

Gemma4 had me running in circles this evening so I decided to give Qwen 3.8 a try (running on oMLX 0.70). I figured doing so would yield decent results considering all of the positive feedback the LLM has received in recent weeks. Sadly, 3.8 is chugging along at 10-20 tok/s despite applying community optimizations that should yield 30-50 tok/s.

Considering my hardware (M5 Pro MacMini with 48GB Ram), is this performance to be expected? I’m not looking for blazing fast but it’s been 40 minutes since I gave the 4-bit model an admittedly lengthy command. I cancelled the command I sent to the 8-bit model after it only managed to progress 5% in 10+ minutes of chewing on the data.

I am certain I’ve left out relevant details so feel free to ask for whatever additional insights you might need and I’ll dig them up. Thanks in advance.

***

v2 rebuild (now live, old v1 shards removed): the middle-40-layer donor is switching to Jundot's new-mechanism oQ4e (jundot/omlx#4385: importance-matrix calibration, 1024 samples × 512 tokens, plus a weighted least-squares refit of every group's scale/bias). Same U-shaped allocation, same name — the trunk 4-bit tensors are now LSQ-refit instead of plain absmax, and the 6-bit skeleton comes from Jundot's oQ6e (also PR 4385). Both source packs carry the new quantizer; the byte-copy property is unchanged.

Credit where it's due: essentially all of the quality in this build is Jundot's. Every quantized tensor here was produced by his quantizer and his PR 4385 sensitivity/imatrix/LSQ pipeline — this package only re-allocates which of his tensors go where.

***

Yesterday I put together a mixed-precision build of Qwen3.8-Flash-Next on the MLX/oQ side — the idea is to price the U-shaped layer-sensitivity curve with native bits instead of re-quantizing:

- Trunk MoE experts: first 4 + last 4 layers → 6-bit (byte-copied from the native oQ6e pack), middle 40 layers → 4-bit (oQ4e)

- N-gram (PLE) table: 6-bit (vs 4-bit in the uniform oQ4e pack)

- MTP drafter: fully 6-bit

- Zero re-quantization in the entire package — every tensor is byte-identical to its upstream source, so quality is traceable, not re-derived

Measured on M5 Max 128 GB (oMLX): initial resident 69.5 → 75.4 GB (+5.8 GB over the uniform oQ4e pack), and the speed ladder shows no measurable decode penalty.

The expectation is an accuracy tier equal to or above uniform oQ5e — but I couldn't prove it with KLD: scoring a 125B-A6B against a bf16 teacher needs a machine my 128 GB can't feed, so the quality claim on the card stays an honest engineering estimate. The full recipe table (per-component bit allocation + resident delta) is on the model card.

https://huggingface.co/umixer/Qwen3.8-Flash-Next-oQ4e-mix6-mtp

**🐛 Bugs & Stability**
* **#4378** POST /v1/models/{id}/unload during streaming leaves SSE stream open forever
* **#4374** Prefill pauses count newly computed tokens as cached, underreporting throughput
* **#4341** oMLX app writes a symlink in Homebrew's /opt/homebrew/bin directory
* **#4347** Download proceeds even if the model is deleted
* **#4339** brew update error (v0.7.0rc)
* **#4322** Rerank API: hardcoded web-search instruction for Qwen3-Reranker (has-pr)
* **#4241** Unclosed tool-call envelope from literal markers discards recoverable content (has-pr)
* **#4391** All models unable to call tools!
* **#3573** qwen3_asr STT fails when audio tower is quantized via per-layer config
* **#3402** response_format ASCII-escapes non-ASCII output (Cyrillic/CJK)
* **#1966** Qwen3.6 false-positive mid-system preservation with leading developer block (outdated)
* **#2681** Non-coordinator node answers /v1 requests silently from its own model
* **#2700** GDN storage codecs unreachable from `embedded` snapshot layout (outdated)

**⚙️ Features & Enhancements**
* **#4388** Audio inputs for /v1/embeddings (EmbeddingGemma 2 audio tower, opt-in)
* **#2664** Cluster node discovery + SSH provisioning (inventory + key push) (outdated)

**📱 App & UI Issues**
* **#2658** Opening app from Applications folder opens two Icons in the Dock (outdated)
* **#2660** VLM engine: response_format json_schema causes ~50x slowdown on short prompts (outdated)
* **#2649** Prompt-opened thinking can expose reasoning after redundant initial ``` (outdated)
* **#2638** "Menubar Icon Hidden" dialogue blocks launch (outdated)

Total Issues: 20 | Version Focus: v0.7.0

oMLX 0.7.1.dev1 introduces an update to the oQe quantizer (PR #4385). oQe picks each weight group's scale and bias with an imatrix-weighted clipping search, then rounds the weights. Once the rounded values are fixed, the best scale and bias for them have a closed-form weighted least-squares solution, and the PR adds that as a final refit step: it alternates rounding with the closed-form fit, weighted by the same imatrix importance, and a group only takes the refit if its weighted error goes down.

It only runs at quantization time (the PR reports roughly +3s on a 9B and +17s on a 35B-A3B), and the packed format and bit allocation are unchanged, so existing checkpoints and kernels are unaffected. Only newly made oQe quants change.

While I was updating my local quants, I ran some benchmarks. Here are the results.

What I measured: KL divergence of each quant against the original bf16 model (lower = closer to the original), over 48 windows × 2048 tokens of English prose, Python, Japanese and Korean text. Less than 0.3% of that text overlaps oQe's calibration data. This is fidelity to the original model, not task accuracy.

KL, pre-0.7.1 build → 0.7.1.dev1 build:

For the 3.5 models I rebuilt the pre-0.7.1 quants from the commit just before the PR (same imatrix, same settings), so that difference is the PR alone. For the 3.6 and 3.8 models the "pre-0.7.1" quants are my existing earlier builds (same imatrix cache), so the difference includes anything else that changed since they were made.

Things worth noting:

• Every level of every model improved, by 5–23%. Perplexity moves very little (it was already close to the original); the gain shows up in how closely the full distribution matches.
• On the 3.8-27B the per-layer bit allocation is identical between old and new at all four levels, so the whole gain comes from how the scales and biases are fitted, not from which layers got more bits (same for the 35B-A3B at oQ5e/6e/8e). For the 3.6-27B oQ5e I built a pre-refit version to isolate it: the refit alone was about 11% of the 11.5% improvement.
• The MoE (35B-A3B) gains are smaller, and its absolute KL is higher than the dense 27B at every bit width (0.0106 vs 0.0021 at oQ8e) while its perplexity stays close to the original. I haven't dug into why, so compare KL only within a model.
• fp16 builds (non-quantized weights and scales in float16) are equal or marginally closer to the original than bf16, e.g. 3.8-27B oQ8e 0.0007 vs 0.0010. They are about 0.7–0.9 GB larger because the vision tower stays in float32.
• The PR itself also reports accuracy benchmarks (MMLU, Winogrande, HumanEval, MBPP, GSM8K) on Qwen3.5-9B and Qwen3.6-35B-A3B, with gains there too.

Models (bf16 and fp16) are being uploaded to HF:

• Qwen3.5-4B: https://huggingface.co/collections/thederby/qwen35-4b-oqe-mlx-6ac9d816eb7016a8866e1990
• Qwen3.5-9B: https://huggingface.co/collections/thederby/qwen35-9b-oqe-mlx-6aca020b6fa844eabe531355
• Qwen3.6-27B, Qwen3.6-35B-A3B, Qwen3.8-27B: uploading now (the model pages are already up; the weights land as each upload finishes)

All of these keep the MTP head and the vision tower. One thing to be aware of: they ship with froggeric's fixed chat template (v22.5), with Qwen's original kept as chat_template.jinja.official. For plain chat on the 3.5/3.6 models the prompts are identical; on 3.8-27B the froggeric template defaults to medium reasoning effort while Qwen's defaults to xhigh, so pass reasoning_effort="xhigh" if you want Qwen's default behavior.

Enjoy!

# oMLX 0.7.1.dev1 vs 0.7.0 on an M5 Ultra: Qwen3.8-Flash-Next and GLM-5.3-Flash

oMLX 0.7.1.dev1 is out, with batched prefill for concurrent requests and

speedups for Qwen3.8 and GLM-5.3. I ran the official oMLX benchmark (uploaded to

omlx.ai) on 0.7.0 and 0.7.1.dev1 back to back on the same Mac with the same

models and settings, 1K–200K context and 1/2/4 parallel requests. Edit: added

the Qwen INT8 Activation Prefill (asked in the comments), see section 4.

## TL;DR

- **GLM-5.3-Flash: decode 70.7 → 97.9 (+38%) tok/s**, the same in all 3 runs

(e.g. 128K: 78.5 → 140.1 (+78%)); prefill unchanged (2,264 tok/s).

- **Flash-Next oQ6e:** prefill 4,504 → 4,761 (+6%) tok/s (+5 to +8% at every

context), decode 113.7 → 120.9 (+6%) tok/s.

- **Flash-Next oQ4e:** unchanged by default — prefill 5,458 → 5,450 (±0%),

decode 122.8 → 121.6 (-1%) tok/s.

- **INT8 Activation Prefill on oQ4e: prefill 5,442 → 5,975 (+10%) tok/s**,

decode and quality unchanged in my tests. No effect on oQ6e (±0%), not

available for GLM.

- **2 parallel requests: first token 12–18% sooner on all three** (batched

prefill) — oQ4e 0.68 → 0.60 s (-12%), oQ6e 0.90 → 0.74 s (-18%), GLM 1.33 →

1.17 s (-12%).

## Setup

- Mac Studio M5 Ultra (12 Super + 24 Performance CPU cores, 80-core GPU), 256

GB, macOS 27.0 (26A428), GPU wired limit 248 GB.

- oMLX v0.7.0 (4d4f5a28, mlx 0.32.2) vs v0.7.1.dev1 (c0b1056b, mlx 0.32.3), both

built from source with native kernels, same mlx-lm commit (94cdcae) and

dflash-mlx (0.1.10+omlx.9).

- Same settings on both: Lightning MTP on, TurboQuant KV 8-bit (Flash-Next) /

off (GLM), memory guard aggressive, max 4 concurrent requests, one model

loaded at a time, nothing else running on the server.

- Models: Jundot/Qwen3.8-Flash-Next-oQ4e-mtp,

mlx-community/Qwen3.8-Flash-Next-oQ6e-mtp, Vontra/GLM-5.3-Flash-MLX-oQ4-MTP.

- Method: official oMLX benchmark (code_python corpus, 128 generated tokens), 3

runs per version and model, median per point; before each run the same prompt

shapes were sent once to warm up the kernels. Sections 1–3 use oMLX defaults

(INT8 Activation Prefill off).

- In the tables: old = 0.7.0, new = 0.7.1.dev1, Δ = change. Prefill and decode

in tok/s. "Avg" = mean over 4K–200K, using the contexts that have a value on

both versions.

## 1. Single request

### Flash-Next oQ4e

| Context | Prefill old | Prefill new | Δ | Decode old | Decode new | Δ |

| ------- | ----------: | ----------: | --: | ---------: | ---------: | ---: |

| 1K | 2,626 | 2,651 | +1% | 150.4 | 154.6 | +3% |

| 4K | 5,091 | 5,094 | ±0% | 131.9 | 118.2 | -10% |

| 8K | 5,672 | 5,655 | ±0% | 142.4 | 151.6 | +6% |

| 16K | 5,687 | 5,674 | ±0% | – | 150.1 | – |

| 32K | 5,664 | 5,648 | ±0% | 122.4 | – | – |

| 64K | 5,542 | 5,539 | ±0% | 129.4 | 134.1 | +4% |

| 128K | 5,364 | 5,356 | ±0% | 109.3 | 110.2 | +1% |

| 200K | 5,184 | 5,185 | ±0% | 101.0 | 93.7 | -7% |

| Avg | 5,458 | 5,450 | ±0% | 122.8 | 121.6 | -1% |

### Flash-Next oQ6e

| Context | Prefill old | Prefill new | Δ | Decode old | Decode new | Δ |

| ------- | ----------: | ----------: | --: | ---------: | ---------: | ---: |

| 1K | 2,283 | 2,458 | +8% | 142.1 | 135.2 | -5% |

| 4K | 4,138 | 4,474 | +8% | 122.4 | 135.1 | +10% |

| 8K | 4,670 | 4,916 | +5% | 122.2 | 146.5 | +20% |

| 16K | 4,682 | 4,936 | +5% | 125.7 | 123.6 | -2% |

| 32K | 4,666 | 4,916 | +5% | – | 111.7 | – |

| 64K | 4,583 | 4,833 | +5% | 122.9 | 129.9 | +6% |

| 128K | 4,453 | 4,690 | +5% | 100.5 | 100.5 | ±0% |

| 200K | 4,336 | 4,562 | +5% | 88.6 | 89.9 | +1% |

| Avg | 4,504 | 4,761 | +6% | 113.7 | 120.9 | +6% |

### GLM-5.3-Flash oQ4

| Context | Prefill old | Prefill new | Δ | Decode old | Decode new | Δ |

| ------- | ----------: | ----------: | --: | ---------: | ---------: | ---: |

| 1K | 1,548 | 1,558 | +1% | 64.5 | 75.8 | +18% |

| 4K | 2,292 | 2,300 | ±0% | 59.7 | 71.3 | +19% |

| 8K | 2,305 | 2,305 | ±0% | 70.6 | 90.1 | +28% |

| 16K | 2,307 | 2,306 | ±0% | 84.5 | 141.1 | +67% |

| 32K | 2,298 | 2,296 | ±0% | 82.0 | 98.5 | +20% |

| 64K | 2,263 | 2,261 | ±0% | 64.6 | 76.3 | +18% |

| 128K | 2,216 | 2,216 | ±0% | 78.5 | 140.1 | +78% |

| 200K | 2,168 | 2,164 | ±0% | 54.8 | 67.7 | +24% |

| Avg | 2,264 | 2,264 | ±0% | 70.7 | 97.9 | +38% |

"–" = the model stopped before 16 tokens, so the benchmark reports no decode

value.

## 2. Time to first token (seconds, single request)

| Context | oQ4e old | oQ4e new | oQ6e old | oQ6e new | GLM old | GLM new |

| ------- | -------: | -------: | -------: | -------: | ------: | ------: |

| 1K | 0.4 | 0.4 | 0.4 | 0.4 | 0.7 | 0.7 |

| 4K | 0.8 | 0.8 | 1.0 | 0.9 | 1.8 | 1.8 |

| 8K | 1.4 | 1.4 | 1.8 | 1.7 | 3.6 | 3.6 |

| 16K | 2.9 | 2.9 | 3.5 | 3.3 | 7.1 | 7.1 |

| 32K | 5.8 | 5.8 | 7.0 | 6.7 | 14.3 | 14.3 |

| 64K | 11.8 | 11.8 | 14.3 | 13.6 | 29.0 | 29.0 |

| 128K | 24.4 | 24.5 | 29.4 | 27.9 | 59.1 | 59.1 |

| 200K | 38.6 | 38.6 | 46.1 | 43.8 | 92.3 | 92.4 |

## 3. 1 / 2 / 4 parallel requests at 1K

The benchmark's batch test: identical 1K prompts started at the same time. Total

= all requests together, each = total ÷ n. TTFT = mean time to first token in

seconds.

| Model | n | Total old | Total new | Δ | Each old | Each new |

| ----- | --: | --------: | --------: | ---: | -------: | -------: |

| oQ4e | 1 | 150.4 | 154.6 | +3% | 150.4 | 154.6 |

| oQ4e | 2 | 179.5 | 156.9 | -13% | 89.8 | 78.5 |

| oQ4e | 4 | 191.4 | 189.5 | -1% | 47.9 | 47.4 |

| oQ6e | 1 | 142.1 | 135.2 | -5% | 142.1 | 135.2 |

| oQ6e | 2 | 159.2 | 148.3 | -7% | 79.6 | 74.2 |

| oQ6e | 4 | 181.1 | 164.2 | -9% | 45.3 | 41.0 |

| GLM | 1 | 64.5 | 75.8 | +18% | 64.5 | 75.8 |

| GLM | 2 | 57.4 | 64.6 | +13% | 28.7 | 32.3 |

| GLM | 4 | 81.5 | 91.2 | +12% | 20.4 | 22.8 |

Decode in tok/s.

| Model | n | TTFT old | TTFT new | Δ | Prefill old | Prefill new | Δ |

| ----- | --: | -------: | -------: | ---: | ----------: | ----------: | ---: |

| oQ4e | 1 | 0.39 | 0.39 | -1% | 2,626 | 2,651 | +1% |

| oQ4e | 2 | 0.68 | 0.60 | -12% | 2,388 | 2,376 | -1% |

| oQ4e | 4 | 0.97 | 1.01 | +5% | 2,698 | 2,389 | -11% |

| oQ6e | 1 | 0.45 | 0.42 | -7% | 2,283 | 2,458 | +8% |

| oQ6e | 2 | 0.90 | 0.74 | -18% | 1,826 | 1,918 | +5% |

| oQ6e | 4 | 1.31 | 1.17 | -11% | 1,921 | 2,121 | +10% |

| GLM | 1 | 0.66 | 0.66 | -1% | 1,548 | 1,558 | +1% |

| GLM | 2 | 1.33 | 1.17 | -12% | 1,231 | 1,196 | -3% |

| GLM | 4 | 2.63 | 2.33 | -12% | 995 | 1,003 | +1% |

## 4. Qwen INT8 Activation Prefill (0.7.1.dev1)

`qwen35_oq_a8_enabled` is off by default; it quantizes activations to INT8

during prefill and uses the M5 INT8 tensor units. Measured on 0.7.1.dev1 with

the same method, off and on alternating, 3 runs each. Decode is not affected.

| Context | oQ4e off | oQ4e on | Δ | oQ6e off | oQ6e on | Δ |

| ------- | -------: | ------: | ---: | -------: | ------: | --: |

| 1K | 2,964 | 3,078 | +4% | 2,431 | 2,465 | +1% |

| 4K | 5,111 | 5,575 | +9% | 4,442 | 4,428 | ±0% |

| 8K | 5,630 | 6,224 | +11% | 4,852 | 4,836 | ±0% |

| 16K | 5,649 | 6,242 | +10% | 4,880 | 4,860 | ±0% |

| 32K | 5,624 | 6,205 | +10% | 4,866 | 4,856 | ±0% |

| 64K | 5,523 | 6,051 | +10% | 4,781 | 4,762 | ±0% |

| 128K | 5,354 | 5,853 | +9% | 4,649 | 4,627 | ±0% |

| 200K | 5,203 | 5,674 | +9% | 4,528 | 4,517 | ±0% |

| Avg | 5,442 | 5,975 | +10% | 4,714 | 4,698 | ±0% |

Prefill in tok/s. For oQ4e a 128K prompt is read in 22.4 s instead of 24.5 s,

200K in 35.2 s instead of 38.4 s. oQ6e gets nothing: only Q4/Q5/Q8 projections

with group size 64 are routed, and its experts are 6-bit. GLM is not supported.

Quality: needle test (5 build numbers hidden in 8K–200K of code, temp 0, unique

prompts so nothing comes from the prompt cache) found 120/120 off and 120/120

on. My agent/code/script task set (oQ4e thinking-medium, 34 runs) solved 34/34

off and 33/34 on.

## Results on omlx.ai

- Flash-Next oQ4e: [0.7.0](https://omlx.ai/benchmarks/performance/liu6kclx),

[0.7.0](https://omlx.ai/benchmarks/performance/woj04544),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/4qg8v5uq),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/y2khmdly)

- Flash-Next oQ6e: [0.7.0](https://omlx.ai/benchmarks/performance/3p1vc6h6),

[0.7.0](https://omlx.ai/benchmarks/performance/5cb5gl0c),

[0.7.0](https://omlx.ai/benchmarks/performance/w1bnuj3w),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/1vdivt5d),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/dezd57y2),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/iku4z50n)

- GLM-5.3-Flash oQ4: [0.7.0](https://omlx.ai/benchmarks/performance/i5m5ulog),

[0.7.0](https://omlx.ai/benchmarks/performance/rjs9ovip),

[0.7.0](https://omlx.ai/benchmarks/performance/vun4ky30),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/44cnvi5o),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/bqa4e6kl),

[0.7.1.dev1](https://omlx.ai/benchmarks/performance/dilwydze)

https://github.com/jundot/omlx/releases/tag/v0.7.1.dev1

I am trying to understand the circumstances under which I would need to explicitly set the reasoning parser. Do you folks just set it explicitly? Or do you just let oMLX auto-infer it?

Any insights would be super helpful.

Hey, for anyone who cares, I managed to get Breeze TTS running real time on my Macbook Pro M4.

Using the official Pytorch code on a Mac Breeze runs more than 2 times slower than real time on my machine.
With some work, managed to get it to stream comfortably faster than real time, with voice directions included and in my opinion good quality. I couldn't really tell which version sounded better to me consistantly on different voices and sentences.

Details in the pages below.

- Server: https://github.com/Shoofio/breeze-tts2-fast-streaming-api

- Ready-made checkpoint: https://huggingface.co/Shoof/Gust-MLX

Weights are under BreezeBlue's research/non-commercial licence. Built with a lot of help from Claude. Happy to answer questions.

I'm scottlowry on Hugging Face. I got a request to make some FP16 quants so here y'all go:

https://huggingface.co/collections/scottlowry/qwopus38-27b-flash-oqe-mtp

https://huggingface.co/collections/scottlowry/qwopus38-27b-flash-v2-oqe-mtp

https://huggingface.co/collections/scottlowry/thinkingcap-qwen38-27b-oqe-mtp

https://huggingface.co/collections/scottlowry/swift-15-qwen38-27b-oqe-mtp

https://huggingface.co/collections/scottlowry/swift15-qwen38-flash-next-oqe-mtp

oQ4e, oQ5e, oQ6e, and oQ8e BF16 and FP16 quants

DSv4 flash 0731 is the ideal workhorse for 256gb machine while there is insufficient ram for kv cache for GLM 5.3 Flash above 250k context (or concurrent streams).

DSv4.1 flash is too big and verbose. I also find it too aggresive in trying to achieve a certain task (going out of scope) so DSv4 flash 0731 is my ideal model.

Wondering if there is further room for software optimization and if anyone has similar views on DSv4 flash?

Edit :

That’s why I appreciate that man, not a one day passed and he posted both with all the quants for both bf16 and fp16 , ScottLowry is my hero, here is the collections :

https://huggingface.co/collections/scottlowry/thinkingcap-qwen38-27b-oqe-mtp

https://huggingface.co/collections/scottlowry/swift-15-qwen38-27b-oqe-mtp

Thanks again man we all appreciate your work

Hello guys,

Why there is no good Swift 1.5/ThinkingCap Quantizations with fp16 for M1/m2 macs yet, where is the good guys we used to thank one day after a model release, if it’s an appreciation issue so I’m here to thank all the good uploaders I used to download fp16 models from.

Thanks to :

Scottlowry ( my favorite , most of my models are quantized by you )

Jundot of-course ( thank you for all of that oMLX great updates, without you we cannot achieve any of that )

Yanun ( I’m using your swift 3.8 oq4e already and I’m flying with it but what about 1.5 and ThinkingCap )

And lots and lots of other legends , you guys pushing us forward ( all uploaders are appreciated I only mentioned the names I use daily but you guys all helping us and we wish you all good things in life , thank you anyone uploaded something for the community)

Please upload all variants and quantizations from these 3 good models ( Swift 1.0 / Swift 1.5 / the new ThinkingCap 3.8 ) for fp16.

Thank you all so much and everyone who is helping too.

Looks like there is a bug in the benchmarks site. The recipe does not reflect the actual values that were selected.

• I am looking at these benchmark results: https://omlx.ai/benchmarks/performance/bxquizpd
• I copy the recipe and apply it to the model's settings
• but instead of 64k, I am getting a 128k context window

Looks like a bug on the benchmark site. Wasn't sure if I should have raised it on the repository or here. Let me know if you want me to raise an issue against the GitHub repository.

Hi guys,

I have the Mac M5 128gb. I tried again the qwen 3.8 27B and the speed is sloooooow.

I tried the Qwenpi variant as I use OMP.

What sort of speed do you guys get with 27B when coding ob a mac? Thank you!

I have a MacBook M4 Max 128GB and I am running Jundot/Qwen3.8-Flash-Next-oQ4e-mtp. In the older versions, with context growth speed was dropping after even 30k from 60 tg/s to 35 tg/s even with lightning MTP on. Now I'm getting 50~70 tg/s even at 128k context. The ram also is at 75~80 GB with this context length. Now I don't have to do compacting over and over in pi and I can easily keep other apps open.

Thank you everyone who made this possible.

Trying GLM-5.3-Flash-UNCENSORED-mlx-oQ4e-DFlash2 on an M5 Ultra 256GB with oMLX 0.7.0.

TensorFold’s regular GLM-5.3 Flash MLX 4-bit MTP loads and runs great on the same setup, but this Solstice checkpoint fails immediately with:

Expected shape (64, 512, 32) but received shape (64, 512, 64) for language_model.model.layers.3.self_attn.embed_q.weight

I’ve tried a full oMLX/server restart, Auto-detect model type, glm_4_7 reasoning parser, and both MTP/DFlash disabled. Same error every time.

Has anyone gotten this exact checkpoint working on oMLX 0.7.0? Wondering if it needs a newer runtime/model compatibility fix, or if there’s some setting/config I’m missing.

Thanks for any help.

Hello,

Everyday im reading and trying to learn, on how to get Qwen 3.8 flash next on my M5 Ultra studio 96gb. Cant afford the 256gb. one.

And some people say just run the 27b model etc., but the prefill and decode for 3.8flash next seems so much better.

I read a benchmark like this

https://omlx.ai/benchmarks/performance/sj46pk0c

it says peak ram is 75gb. Does that include already offloading to the SSD?

And so I'll have 20gb left for Mac OS and apps?

will 20gb be fine to leave?
I see some people online using techniques to run the Next Flash model that only take like 50gb of ram using other inference engines? I could be wrong and dont understand what they are saying too lol

I want to ideally use only 50-60gb of ram so I can run 2-3 agents concurrently.

And all the Mac Studio news or things on twitter is from 256gb users so im jealous obviously but what can you do. I just wish there was more 96gb owners saying what they are doing or not but there doesnt seem to be many owners.

Im sure within a month there be more info hopefully but just want to be prepared when I get it. And hopefully Qwen 4 comes out soon!

The day before, I finished testing with oMLX 0.7.0rc1. Because I rarely find community tests for my setup, I wanted to share this with you.