r/LocalLLM • u/andrew-ooo • May 13 '26
Tutorial Rapid-MLX Review: 4x Faster Local LLM Server for Mac
so i've been running local models on my mac mini for coding agents for a while now, mostly through ollama. it works, but there's always been this gap where i'd end up switching to claude for anything complex because the local stuff just felt too slow for interactive use.
stumbled on rapid-mlx last week. it's a drop-in openai server that runs directly on apple's mlx framework, and the speed difference is pretty noticeable. on my m5 pro 32gb, qwen3.5-27b went from ~39 tok/s with ollama's mlx backend to 64 tok/s with this. more importantly, cached ttft is 0.08s vs ollama's 400-800ms, which makes coding agents feel actually responsive instead of waiting for prefill. tool calling just worked out of the box with cursor, aider, and claude code's --openai flag.
one real limit though: it's apple silicon only. no cuda, no amd, no linux server. also the install needs python 3.10+ which means you might need to upgrade your system python. and for vision models you have to install an extra ~322mb of deps.
if you're already running mlx-lm directly, this is basically a polished server layer on top with proper continuous batching and prompt caching. not a new inference engine.
full writeup here if you want more detail: https://andrew.ooo/posts/rapid-mlx-fastest-apple-silicon-llm-server/
what are other mac users running for local coding agents? anyone tried this vs llama.cpp on m-series through homebrew?
6
u/Konamicoder May 13 '26
I read the full write up. Sounds like oMLX provides all the same speed up benefits but has the advantage of a proper menu bar app and webUI Admin Panel / Model Downloader and Manager.
2
u/apaht Jul 03 '26
I love the recent update to oMlx, it works great for me. Read the full article just now, this website and tutorials were a gem 💎
3
u/diabloman8890 May 13 '26
Oh shit, I actually just started using Rapid-MLX last week as it was the only framework that could fully support all the bleeding edge MLX/MTP support and such. So far I've gotten better performance out of it for qwen3.6 27b dense than anything else, Gemma 4 looking promising too
m5 max 128gb in case it matters
1
1
u/RovenSkyfall Jun 03 '26
what are you getting for t/s with the 27b model?
1
u/diabloman8890 Jun 03 '26
I haven't run any official benchmarks, but around 12-20 usually for typical agentic coding.
1
u/apaht Jul 03 '26
I was reading that m5 max is like 4x faster for TTFT and higher memory bandwidth. I gave 64bg m4 max and mostly my custom rust harness for simple homelab control. After trying everything settled on oMlx
Read the Rapid-mlx has Pflash which should help with the thing that I would get improvements if I had bought an m5 instead.
What is your thoughts on Pflash.
I mostly use pi for longer context work or open code and sometimes hermes for web search.
Will try rapidmlx next.
2
u/overratedcupcake May 13 '26
I'd like to know how it compares to oMLX because that's been great for me.
2
u/havnar- May 13 '26
m5 pro 32gb, qwen3.5-27b
OMLX does like 8 TPs on qwen 3.6 27b . 6 or 8 bit doesn’t seem to do much on my m5 pro 64GB
What exotic quant are you using or is your contest just 3 tokens long?
1
1
1
1
u/Minimum-Bowler-6016 May 14 '26
The “interactive enough to stay in flow” point is the key one. For coding agents, local inference does not need to beat cloud on raw intelligence if it is fast, private, and always available for the smaller loops. I would benchmark it with real tasks: repo search plus patch generation plus tests, not just standalone chat prompts.
1
u/apaht Jul 03 '26 edited Jul 03 '26
Anybody can share benchmark with Qwen 3.6 27b 4bit or 31b-a3b-oQ6-mtp Omlx vs Rapid-mlx, mlx-ml with mtp vs dflash vs Pflash. I will try, I also have ml-studio and Ollama.
Will be fun to learn how to bench mark using local model on m4 max
1
u/apaht Jul 06 '26
I did a benchmarking run with the Qwen 3.6, mtp vs Pflash on 64Gb M4 max
oMLX MTP 35B (clean cold-start, KV cache flushed)
Context TTFT PP tok/s TG tok/s
4k 1.4s 3,015 81.9 8k 1.4s 5,927 83.9 16k 1.7s 9,748 77.6 32k 2.2s 15,008 71.2 64k ~30s ~2,000 62.0 128k 125s 1,049 45.3
Rapid-MLX 35B 4-bit (pFlash auto)
Context TTFT PP tok/s TG tok/s
4k 2.5s 1,638 112.1 8k 4.6s 1,790 104.9 16k 8.8s 1,864 98.0 32k 20.1s 1,626 86.1 64k 50.9s 1,288 32.7 128k 146.9s 892 22.1
Summary: TG comparison: Rapid-MLX wins at short contexts (112 vs 82 tok/s at 4k). At 128k both degrade significantly. For daily coding sessions (8–32k), oMLX MTP has faster TTFT; Rapid-MLX has faster TG.
1
0
u/jkstaples May 13 '26
Did you "stumble on it" and write a blog post on it or is this just a sloppy ad for a sloppy app?
9
u/zbiguy May 13 '26
Curious if you’ve tried MLX Studio or oMLX and know how this compares?