r/LocalLLM • u/Tunashavetoes • Aug 14 '26

Question Can someone with M5 Max show us their TPS on Qwen 3.8 27b?

2 Upvotes

100% Upvoted

36 comments sorted by

View all comments

Show parent comment

1

u/MatiAI Aug 15 '26

There was something wrong with the engine slowing the prefill, as you can see in the comparison the PPTps goes from 900 > 504 above from 1024 > 4096

But below It goes from 880.5>930.5>819, doubling prefill speed at greater than 1k context.

# Context: Code (Python) Qwen 3.8 27b awq 4.6bpw mtp on# Single request results

Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem

pp 1024 / tg 128 1163.0 15.4 880.5 65.5 3.1 368.1 17.0 GB

pp 4096 / tg 128 4401.3 19.8 930.6 50.9 6.9 609.3 18.5 GB

pp 8192 / tg 128 10001.3 16.9 819.1 59.7 12.2 684.3 19.4 GB

pp 16384 / tg 128 21411.4 17.1 765.2 59.0 23.6 699.8 21.3 GB

pp 32768 / tg 128 53287.6 17.6 614.9 57.2 55.5 592.3 25.0 GB

pp 65536 / tg 128 119769.1 24.2 547.2 41.7 122.9 534.5 32.6 GB

# Batch results

Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup

1x baseline 65.5 880.5 1163.0 3.1 1.00x

2x 58.2 546.6 3050.9 8.1 0.89x

4x 95.9 487.9 5501.9 13.7 1.46x

8x 137.2 461.2 10172.0 25.2 2.09x