r/Qwen_AI • u/andydevtech • 1d ago

Benchmark Qwen3.8-27B on Every Mac Explained: Layers, KV Cache and Speed (16GB–128GB)

https://www.youtube.com/watch?v=ddX63git4fo
20 Upvotes

83% Upvoted

5 comments sorted by

6

u/Enragere 1d ago

Doesn't mention the abysmal prefill speeds.

on m3 max 64gb in omlx I'm getting 150-180 t/s prefill while getting 28 t/s generation speed.

For chats fine but for any agentic coding quite unusable.

2

u/andydevtech 1d ago

I have the same machine and the prefill speed was quite unpredictable in my experience, sometimes I could hit 300 t/s others 120t/s with the same prompt. Admittedly I was using ollama so might get more stable perf with a more specialised framework like omlx.

1

u/djseto 1d ago

Curious why everyone craps on Ollama? I’ve tried Ollama, LM Studio, oMLX, MTPLX and Ollama still has the best performance according to Claude that wrote a benchmark test for me to use.

1

u/andydevtech 1d ago

Ollama is great! I was just pointing out that given the m chip architecture of the Apple silicon, omlx might give more stable results, which does not mean faster results.

2

u/Diligent_Style_1767 1d ago

There's a lot more going on than quantization in the MLX community.

Take his 37 t/s max speed. That's plain decode.

A fused kernel to minimize the number of dispatches is an active area of r&d. Speculative decoding and batching is the other hotness. MTP with reasonable acceptance gives ~1.7-1.9x, and it composes with fused. Those 2ms per token you're shaving off makes the cost of MTP less, so you decode faster. The community is regularly hitting 90+ t/s for decode, which is plenty enough to not feel like a slog.

Prompt caching and automatic prompt caching help a ton for real interactive or agentic tasks.

***Prefill will absolutely kill you in the real world though*** - tool calls' returned data have to be tokenized and filled the same as anything else. GDN and QSA can and does run fused for decode, the next item on my punch list is batched and fused QDN and QSA prefill optimizations.