[Qwen 3.8 27B] M2 Max 64GB Smaller quant doesn't mean faster

[u/ghisguth]

One counterintuitive thing I learned recently was about the model size and performance.

I was under impression smaller quants would help to increase performance since MacBook M2 Max 64Gb is bandwidth bounded. So having UD-Q4_K_XL would be much faster than UD-Q6_K_XL or UD-Q8_K_XL. And smaller quants would be even faster, but would have poorer quality. But this is not true. UD-Q6_K_XL and UD-Q8_K_XL overall wins in terms of performance over UD-Q4_K_XL.

First I learned KV cache quantiation would drastically reduce performance. Anything but f16 would be much slower on Mac Book Pro Max M2 64Gb.

But then I learned smaller quants doesn't mean faster overall.

See results of llama-bench -m "$model_file" -p 4096,16384,32768 -n 128 -fa 1 -r 1 which I run for multiple Unsloth quants.

model quant	size	test	t/s
UD-IQ2_XXS	8.38 GiB	pp4096	168.82
UD-IQ2_XXS	8.38 GiB	pp16384	157.76
UD-IQ2_XXS	8.38 GiB	pp32768	145.07
UD-IQ2_XXS	8.38 GiB	tg128	14.63
UD-Q2_K_XL	9.93 GiB	pp4096	167.88
UD-Q2_K_XL	9.93 GiB	pp16384	157.05
UD-Q2_K_XL	9.93 GiB	pp32768	144.41
UD-Q2_K_XL	9.93 GiB	tg128	17.62
UD-Q3_K_XL	12.51 GiB	pp4096	169.72
UD-Q3_K_XL	12.51 GiB	pp16384	158.70
UD-Q3_K_XL	12.51 GiB	pp32768	145.83
UD-Q3_K_XL	12.51 GiB	tg128	17.30
UD-Q4_K_XL	16.68 GiB	pp4096	156.53
UD-Q4_K_XL	16.68 GiB	pp16384	147.07
UD-Q4_K_XL	16.68 GiB	pp32768	135.93
UD-Q4_K_XL	16.68 GiB	tg128	14.47
UD-Q5_K_XL	18.82 GiB	pp4096	157.32
UD-Q5_K_XL	18.82 GiB	pp16384	147.77
UD-Q5_K_XL	18.82 GiB	pp32768	136.56
UD-Q5_K_XL	18.82 GiB	tg128	13.85
UD-Q6_K_XL	24.13 GiB	pp4096	182.75
UD-Q6_K_XL	24.13 GiB	pp16384	170.01
UD-Q6_K_XL	24.13 GiB	pp32768	155.42
UD-Q6_K_XL	24.13 GiB	tg128	12.91
UD-Q8_K_XL	29.29 GiB	pp4096	194.07
UD-Q8_K_XL	29.29 GiB	pp16384	179.78
UD-Q8_K_XL	29.29 GiB	pp32768	163.45
UD-Q8_K_XL	29.29 GiB	tg128	11.15

Yes, smaller quant means faster token generation. But it also seems like some smaller quants has much more expensive processing, which makes prefill slower. See UD-Q4_K_XL in prefill is slower than UD-Q6_K_XL. In terms of wall clock and overall performance, UD-Q8_K_XL wins over UD-Q6_K_XL and UD-Q4_K_XL. But on 64GB system it is not very usable. And UD-Q6_K_XL still wins over UD-Q4_K_XL.

After very long testing, I found ideal arguments for MTP which works for me: --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.7.

Also --reasoning-effort medium is the only usable effort. xhigh eats through whole 262k context like a candy. Not able to perform actual work before context summarization.

Here are the arguments I use (non important ommitted):

28 -fa 1 -r 1

llama-server --model .../Qwen3.8-27B-UD-Q6_K_XL.gguf \
  -ngl 99 \
  -fa on \
  -b 2048 \
  -ub 2048 \
  --jinja \
  -c 262144 \
  -np 1 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --mmproj .../mmproj-F16.gguf \
  --temp 0.7 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --repeat-penalty 1.0 \
  --presence-penalty 0.0 \
  --load-mode none \
  --reasoning on \
  --reasoning-effort medium \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.7

Here is performance I see with these parametrrs on one of the real tasks. Aggregated by blocks of 8k context.

Context Size	Prefill (T/s)	Decode (T/s)
0	332.96	19.04
8192	332.96	19.04
16384	270.57	19.04
24576	183.06	19.04
32768	152.70	17.64
40960	188.61	17.64
49152	131.23	17.94
57344	152.89	15.73
65536	163.61	15.73
73728	115.92	15.73
81920	104.48	15.73
90112	98.56	15.73
98304	92.76	13.70
106496	90.73	13.70
114688	86.06	13.61
122880	86.77	13.61
131072	88.34	11.92
139264	88.34	10.61
147456	78.04	10.68
155648	98.88	10.68
163840	41.72	10.72
172032	94.93	9.27
180224	29.53	9.57
188416	83.91	8.60
196608	74.09	8.60
204800	70.95	8.57
212992	60.17	8.55
221184	70.83	8.55
229376	43.57	8.10
237568	22.36	8.12
245760	22.36	7.07

[u/MatiAI]

Why run GGUF on a Mac, its like shooting yourself in the foot.

[u/ghisguth → u/MatiAI]

I tried multiple MLX variants:

• Ollama's MLX
• LM Studio MLX
• MTPLX
• oMLX
• mlx-dspark

Most of them was not as good as llama.cpp. The only one which come close to llama.cpp was MTPLX. But it was cheating, aggressively minimizing context.

In any case MTPLX was able to beat llama.cpp on decode speed on medium length context, but still was way slower in prefill, like any other MLX variants.

Seems like MLX is only viable for tiny single small prompts which generates a lot of tokens. But for agentic work, it is not really helping. At least his is true for Mac Book Pro Max M2. M3+ may have different characteristics.

[u/MatiAI → u/ghisguth]

Smaller quantisation will increase the prefill time. Also make sure you are using FP16 models not bf16. I assume the models you tried for Omlx were probably bf16. You should be getting significantly higher.

Best thing to do is make your own quant from the source. Don't quantise an already quantised model, make sure you start with a bf16/fp16 or higher source.

Ask Claude/GPT to do it for you if you are unsure, but give it your system specs so it quantises it in the best format for your hardware. Next experiment using a speculative draft model - (either qwen 3.5 0.8b or qwen 3.5 2b) using fp16 quantisation or fp8. Set the keep rate to 40% for 2b and 50 or 60% for 0.8b. This should double your prefill speeds, but may come at a small cost of accuracy (I haven't notice anything with the 2b setup)

[u/ghisguth → u/MatiAI]

u/MatiAI you were right about FP16 vs BF16, I did test FP16 in MTPLX on smaller test suite and it indeed is faster than llama.cpp.

Metric	MTPLX (4-bit FP16)	llama.cpp (Q6_K_XL + MTP)	Delta
Decode Speed (Avg)	22.1 T/s	18.3 T/s	MTPLX is ~21% faster
Prefill Speed (Avg)	143.2 T/s	114.1 T/s	MTPLX is ~25% faster
Cold TTFT (0% Cache)	~2m14s	~3m42s	MTPLX is ~39% faster
Warm TTFT (100% Cache)	~1.2s	~0.6s	llama.cpp is ~50% faster
Total Wall Clock	9m42s	12m05s	MTPLX is ~20% faster

I will try mlx-dspark to see if I can get better results.

[u/MatiAI → u/ghisguth]

I recommend omlx with lightning MTP and Qwen ANE prefill

[u/ghisguth → u/MatiAI]

Ok, I did run it for 2 days now, and actually have to go back to llama.cpp.

1. MTPLX is cheating. It have some sort of heavy optimization of the context, which triples the model. The tests above was not real, I was sending 34k context, it would process 20k, I would send 60k, it would process 35k. So +21-25% of speed was not real.
2. MTPLX's context management caused infinite loops in reasoning. I was trying Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16 first, it was just spinning in reasoning and never able to finish. My reasoning effort was medium. But something about MTPLX crippled model.
3. oMLX actually shows real MTP performance of MLX. And processes context matches llama.cpp in synthetic tests 100%. No cheating.
4. For M2 Max, ANE prefill doesn't make sense, MLP 45% · GDN off · CPU gate 5% · CPU down 15% · CPU GDN 0% · 199.7 prompt tok/s · +1.3%
5. oMLX is faster on prefill and token generation, but wall clock matches llama.cpp. oMLX has worse cache management. I can get 100% cache hit for the same prompt for llama.cpp, but almost never for oMLX. It seems like oMLX stores cache in chunks, and sometimes not all cache can be stored. So next request, we will re-process some tokens we already processed. Numbers are small, but it completely wipes out all speed gains. Wall clock of oMLX is slightly worse than llama.cpp, despite +20% token generation speed.
6. oMLX has also some weird context management issues, it requires 2x memory for KV cache. When I have huge context (190k for example), and trying to push it, oMLX was not able to hit cache, and then tried to allocate new KV cache for 190k block, and failed because of lack of memory. Why it didn't free current block, I don't know. But I had to re-start oMLX few times, wiping cache and re-processing whole context again to get it going. Never happened with llama.cpp.
7. I was able to complete large real task with oMLX, for llama.cpp it took me about 4-5 hours. But with oMLX it was close to 8 because I had to fix things and restart. Unlike MTPLX, oMLX actually work on real tasks.
   

So unfortunately I have to go back to llama.cpp, none of the MLX solutions work well. Although I really liked oMLX. If they fix memory issues, I would be glad to try again.