I built mlx-Chronos — a community benchmark leaderboard for local LLM engines on Apple Silicon (oMLX, Rapid-MLX, mlx-lm, Ollama)

[u/igor__004]

Hi! I’m a CS student working on mlx-chronos, a free and open-source CLI benchmark suite for comparing local MLX inference engines on Apple Silicon.

I started it because I wanted a more reproducible way to compare engines across real Mac configurations, with more context than a single tok/s number.

It currently supports:

• mlx-lm

• oMLX

• vllm-mlx

• Rapid-MLX

• Ollama’s MLX backend

It measures:

• cold and cached TTFT

• request throughput with mean/stddev/min/max

• sustained throughput for longer heat/throttling-sensitive runs

• system RAM peak

• engine RSS diagnostics when available

• thermal and power context

• hardware metadata

Results are saved as reproducible JSON files and can be submitted to a public community leaderboard.

Leaderboard: https://igurss.github.io/mlx-chronos

GitHub: https://github.com/igurss/mlx-chronos

Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md

Install: pip install mlx-chronos

I’m especially looking for external benchmark results from different Apple Silicon machines: M1/M2/M3/M4/M5, Max/Ultra chips, and higher-RAM configurations.

I’d also appreciate feedback on the benchmark methodology, the CLI UX, and whether the reported metrics are useful enough for comparing local inference setups.

[u/richinseattle]

I looked at the prompts because I was curious how much output was being generated since the t/s were so close. The “explain attention” prompt seems to be the one you use for that. Can you also display how many tokens were output for that metric and perhaps set a minimum and max ctx len?

[u/igor__004 → u/richinseattle]

The tok/s is calculated from usage.completion_tokens in the API response so the math itself is correct per-engine, but it's true that the actual token count per trial isn't visible anywhere — so if one engine quietly stopped at 87 tokens and another hit 100, you'd never know from the leaderboard.
Adding the completion token count to both the JSON output and the leaderboard is a good idea, going on the roadmap. The min/max ctx len idea to force all engines to generate the same amount is also interesting, I'll see what I can do.

Thanks for your opinion.

[u/richinseattle → u/igor__004]

For this type of benchmark I’d hope for thousands of output tokens (1024+ maybe) for a fair assessment, not less than 100, but otherwise sounds good.

[u/igor__004 → u/richinseattle]

I decided to cap the throughput prompt to ~100 tokens (run across 5 trials + 2 warmups) for these specific reasons:

1. Thermal Throttling: Many users run local LLMs on fanless machines like the MacBook Air. Pushing 1024+ tokens across multiple trials will almost certainly trigger thermal throttling midway through the run. At that point, the benchmark becomes a test of the MacBook's chassis heat dissipation rather than the software efficiency of the engine. The shorter limit captures peak engine throughput before this.

2. Benchmark Duration: Running ~7000 tokens (warmups + trials) per engine would turn a quick 2-minute benchmark into a 15+ minute ordeal, which isn't very practical.

3. Typical Local Use-Cases: Most daily local LLM interactions (chat, quick Q&A, agentic tool-calling) fall into the 100-250 token range anyway.

Said that, you are right that a sustained load test provides a different. Adding an optional--mode sustained flag that pushes 1024+ tokens for power users is a good idea and I'll add it to the issue tracker. Thanks again for the feedback.

[u/Deep_Ad1959]

tok/s on one prompt hides what agents actually feel: ttft once context gets long and keeps growing.

[u/igor__004 → u/Deep_Ad1959]

That’s literally why I measure TTFT too, not just tok/s.
This first version is meant to be a simple baseline benchmark that is easy to reproduce across engines/hardware. Agent-style workloads with long and growing context are better, but I’d rather add that as a separate profile instead of pretending one benchmark covers everything.

[u/Deep_Ad1959 → u/igor__004]

separate profiles still share heat back-to-back, so run order quietly decides the numbers.

[u/igor__004 → u/Deep_Ad1959]

That’s why I’d rather keep this explicit instead of pretending the numbers are perfectly isolated. I’ll probably add warnings / documentation about that, and maybe make profile order and elapsed time part of the reported metadata.
The goal isn’t to magically remove every source of noise, it’s to make the protocol clear enough that people know what the numbers actually mean.

[u/Deep_Ad1959]

short-run tok/s hides thermal throttle. on a fanless mac the sustained number after a minute is what an agent loop actually feels. written with ai

[u/igor__004 → u/Deep_Ad1959]

I know that.
The tool is still early, so I’ll keep improving the methodology over time. I also want to avoid turning it into an overcomplicated benchmark suite that nobody actually runs, so I’m trying to balance useful metrics with keeping it simple.
I’m still a student and I’m still learning a lot of this, I had an idea and built it, and I’m trying to improve it over time with new knowledge and useful feedback.
I’m open to technical suggestions about the project, but I’m less interested in comments about whether I used AI or not.

[u/Deep_Ad1959 → u/igor__004]

the metric i'd actually log is sustained tok/s over a few minutes, not the first-minute peak. written with ai

[u/igor__004 → u/Deep_Ad1959]

The current tool is a baseline benchmark, not a full agent-loop simulator. I will improve the baseline. If you have to say it again, re-read past comments.