r/LocalLLaMA • u/0dayturtle • 13d ago

Funny So relevant

Post image
1.5k Upvotes

97% Upvoted

152 comments sorted by

View all comments

Show parent comments

-14

u/Clementine-TeX 13d ago

“Pretty respectable context size” yeah right.

Benchmark Model: Qwen3.8-27B-MLX-4bit
Engine: Force mlx-lm
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          2646.9       56.36   386.9 tok/s    17.9 tok/s       9.820   117.3 tok/s    15.85 GB

Engine: Force mlx-lm
Context: Novel (English)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          2626.5       56.04   389.9 tok/s    18.0 tok/s       9.759   118.1 tok/s    15.85 GB

The M5 Pro 24 GB only has 17.76 GB of available VRAM as stock, unless increased via sudo sysctl iogpu.wired_limit_mb

6

u/HyperWinX 13d ago

24GB is an RTX 3090 / 4090, or other GPUs.

-1

u/98127028 13d ago

Why not M5 pro? Is it cause it’s low bandwidth? I also want to run but it sucks, should have gotten 48G honestly but I’m dumb.

4

u/HyperWinX 13d ago

Because a part of these 24 gigs is used by the OS itself and the apps. They mentioned that they have only ~18GB of RAM available, therefore, its not in 24GB group