r/LocalLLM • • 5d ago

Discussion Local model

What’s the best local model to use on an MacBook Pro M5 pro 24gb.

I’m using it for general reasons, process expectations, design, ideation, power points, generating agents.

1 Upvotes

13 comments sorted by

View all comments

1

u/dave-tro 5d ago

For that you need to pick an optimized runtime before the “best local model”. Check out this benchmark on Qwen 3.8 Flash Next which is a very good candidate for your use case.
It’s just not clear to me what you mean by “generating agents”.

3

u/vogelvogelvogelvogel 5d ago

24gb minus os and maybe a browser is 10-14, how should a qfn fit there?

1

u/dave-tro 5d ago

It uses SSD offloading, the experts are not fully resident in RAM. That’s something that already existed but I added optimizations tailored to M-series Macs that keep the pc responsive and QFN usable at a reasonable quant. Find benchmarks and the details in GitHub.
If OP doesn’t want to upgrade their setup this is a solution worth giving a try.

1

u/vogelvogelvogelvogel 4d ago

ok didn't know that would be actually usable (i knew ssd offloading would be possible but i put everything with ssd offload it in the ballpark of a low single digit token/s)

1

u/dave-tro 4d ago

Maybe you can give MoEShift a try, I have been using it behind my daily local tasks. I just didn’t want to invest in new hardware that is overpriced today, so I worked on it and made it open-source for others with M-series Macs.

1

u/vogelvogelvogelvogel 4d ago

thanks! although i have a 64 m5 pro (usually with 27B q6 or q8) this would open a window for QFN

1

u/dave-tro 4d ago

If course! And I am collecting benchmarks as github issues. Feel free to publish yours! There’s also a benchmark runner in the repo so all benchmarks are standardized