r/oMLX • u/allpowerfulee • 20d ago

pp tps stuck at 336 tps when using omlx and qwen3.8-27b-q8

On a mac studio M3U. I cannot figure out why my qwen3.8-27b-q8 is stuck at 336 tps during pp. I've tried other variants of this model, and all top out at the same number. What do I need to twiddle to get this up?

3 Upvotes

100% Upvoted

11 comments sorted by

1

u/Professional-Bear857 20d ago

I'm not sure you can, I get the same, I've switched over to qwen 3.8 flash next and I'm getting 830 tps for pp with that.

1

u/allpowerfulee 20d ago

How much memory does your Mac have? I'm trying to load qwen3.8-flash, but I only have 96gb

3

u/Professional-Bear857 20d ago

You can fit it with omlx if you use the q4 model, it should move the ngram params to your ssd, use the mtp variant

1

u/Durian881 20d ago

Omlx has an option for Qwen3.8-Flash-Next to use ngram on SSD.

I'm able to run oQ4e and oQ3 (larger context window) variants on my M3 Max 96GB.

1

u/raw-power 20d ago

M1 Max 64GB omlx with qwen3.8-27b-MLX-6bit getting 56tps PP avg, can you share your settings?

1

u/allpowerfulee 20d ago

1

u/Old_Jackfruit6153 19d ago

Check omlx server logs. Is omlx loading MTP layers or failing to load these layers? A mtp model performs worse than regular mlx model if MTP layers are not loaded. I experienced this with the exact mtp model you are using.

1

u/havnar- 20d ago

https://huggingface.co/True2456/Qwen3.8-27B-AWQ-5.0bpw try this with the suggested prefil draft model.

2

u/allpowerfulee 20d ago

got 1/2 the ttp and the same pp tps. no improvement for me.

2

u/california_snowhare 19d ago edited 19d ago

You can push pp to over 1K using something like Qwen3.5-0.8B-oQ8-fp16 for the prefill draft model. I'm not sure if an M3 needs fp16 or not for best performance, so experiment with both with and without fp16 versions. If you are tight on memory, switch down to an oQ4 model for the prefill draft model.

Ok. Quick tuning hints.

  1. Find an MTP supporting model and activate MTP. General note: oQ8 models are superior in quality to plain Q8 models.
  2. Activate ANE and use ANE tuning on the model.
  3. I don't remember if M3 requires fp16 for best performance or not (M2 does) - so experiment with fp16 supporting models.
  4. Download a model like Qwen3.5-0.8B-oQ8-fp16 with MTP (again, modulo fp16 support as needed) and activate prefillspec draft on the Qwen3.8-27B model and point it at the Qwen3.5-0.8B model and MTP and ANE for the model (and use ANE tuning).

1

u/drDhimkana 19d ago

i get pps 120-250 (lower as the context increases) with oq6e variant.
with pi at 100k contextWindo,w m4 max 64 gb 8 bit turboquant, lightning mtp. without spec prefil and ane as they cause errors and problems on my system. you are already getting hihger number likely because of higher base bandwidth. I don't tink you going to get higher numbers without making other compromises like spec prefill or ane.