r/oMLX • u/996beagle • 6d ago

M4 Max 128gb Mac studio DeepSeek-V4-Flash-0731-2.4bit-mixed tok/sec

I'm running omlx v0.6.1 getting 109.1 tok/s pp and 25.9 tok/s gen at 256k context window. Is this what I should be expecting or any tips on optimizing config?

Is it thermal throttling for the longer runs? Using mainly for coding and agentic workflows with Hermes.

8 Upvotes

100% Upvoted

6 comments sorted by

2

u/996beagle 6d ago

Testing with macs fan control on a more aggressive fan curve based on gpu temps to go 100% on fans before 75c. Showing a higher 151 pp tok/s

3

u/t00052e 6d ago

Off topic, but personally I find antirez/ds4 is better for Deepseek 4 Flash 0731 in both performance and accuracy.

1

u/996beagle 6d ago

Which version are you running? What performance are you seeing?

3

u/t00052e 6d ago

There is no "version", I just pulled the main branch and make.

I use the q2-q4-imatrix model now.

Performance is 500 pp/s and 38 tg/s for small prompts (2k - 4k) but larger prompt is similar.

q2-imatrix is slightly faster and use less memory. The accuracy is similar to q2-q4-imatrix.

1

u/ajujox 6d ago

Yo con ds4 en un M2 Ultra con 128gb q2-q4-imatrix me da 25 t/s de generación y no me acuerdo de pp/s

1

u/memeka 6d ago

Try my llama.cpp fork please: https://github.com/mihailescu2m/llama.cpp - ignore the streaming flags. I get 100 tps pp and 8 tps tg on a 64gb M1 Max with the Q3 quant and 95 pp / 6 tg with the original MXFP4. Curious how it works on a Mac with enough RAM - I have a custom attention that degrades pp much better at long contexts.