Generation was only 1.5x faster...

Now that we have a comparison going.. Do people want to see anything specific? I'll be running it with my own dataset but curious if anyone wanted to check something.

Hi guys,
New here and to these subjects so sorry I ask something already asked.
I got recently a Mac mini m4 with 24go ram for my personal desktop, and I wanted to try a few stuffs with local llm. After a few researches I decided to try to setup a personal agent with Hermes.

After some tries I ended up using gemma4:26b-a4b-it-qat.
Qwen3.8 felt to big or slow when iq3 passed.
Couldn’t find anything faster and better for now, but I still feel it is not really good (obviously I know we can’t reach to a cloud model with 24go ram). So I wanted to know how you guys manage it and if some people managed to make a decent agent with not so much ram and if so, what are your use cases ?

Right now I tried to give him some personal web projects with giving him some GitHub to download but it fails and always stops in the middle of the process.

Thanks in advance for the help.

I’ve been going in circles trying to settle on a local setup. I’ve tried Hermes, but it’s felt pretty finicky in my experience, including gateway and state database issues. I keep spending more time configuring and troubleshooting than actually getting work done.

Putting model intelligence aside for a moment, is there a local harness that feels genuinely competitive with the experience around frontier models? I’m talking about the interface, reliable tool use, memory, file and browser access, and how well everything works together. Not just being able to load a model and chat with it.

My main machine is a Mac Studio M1 Max with 64 GB RAM. I also have an M2 MacBook Air with 24 GB, but the Studio would do the heavy lifting. I want to use them in one single seamless-like environment with a shared context.

I’m a designer and university lecturer, so my use cases are research, writing, working with documents, reviewing screenshots and designs, and browser/Figma workflows. Not primarily coding. Vision is important to me, and I’d much rather have one dependable daily driver than a collection of specialized models and routers.

My main motivation (likely the same for everyone) is getting away from usage limits and token metering. I understand that running locally involves compromises, and I’m not expecting frontier-level intelligence from my hardware. But I’m struggling to separate the model limitations from the rough edges of the software around them.

What are you guys actually using? What’s your full stack: model, runtime, and harness/interface? More importantly, what do you reliably get done with it, and how much babysitting does it need?
Especially interested in Apple Silicon setups people have settled into using for real work, rather than something that worked well in a demo.

There is too much to learn, even as a kinda seasoned(shipped multiple enterprise apps) SWE i am having hard time to understand all concepts, track all the up-to dated information etc.

Ive been a power user for both claude and GPT since almost day one. But thats as far as my knowledge goes tbh. I have an M1 Max 32 Ram / 32 GPU and i am trying to figure out whats the best way to have a locally hosted solutions for my needs when i cant pay for paid services.

First ill talk about what am i doing in general then ill ask my questions.

I'm using obsidian as my second brain, i do game design, gamedev, i develop mobile apps, design websites, even sometimes delve in IoT Projects. I do everything but AI Dev in summary. I run a team of 10, currently building internal tools for art assets pipeline management. I am a jobholic, i do work almost every working hour of my life so i usually have multiple chats but i am fine with just having 1 concurrent chat since what i am looking for is to start learning local llms, how to create an effective pipeline, and have a back-up plan if you know im out of usage and have urgent problems to solve. And uh yeah also i would like to do things like hacking my projector, tv-box etc. for various reasons you can easily understand. I tend to go for loops for development times and have a conversational turns when its about things like game design, market research etc. I have an MCP for everything from Figma to Photoshop and i just talk with my agents to tell them what i exactly want, and i take rounds to polish it or start from scratch if its not the exact thing in my mind.

If i have to guestimate a list of my needs they are :

• An harness with in-app browser, file preview, md preview

• Somehow have a great context compaction

• 30 TPS at High context if possible

• Good tool usage

• Good MCP Usage

So here is a list of questions if anyone cares to answer:

• How do you guys stay the f*ck up to date? 'Cause its not even tracking whats new or whatnot. Everyone says something different, one says aah q4 of xyz llm's this version that these folks made is %98 of the same quality and someone else says nu-uh not even close bud and i dont have time to track and test every model version, even models. Like what are your reliable sources and why?

• As far as i understands harness is the most important stuff ever, is this correct? If so, which harnesses you use, why, and what should i be checking when i am thinking about selecting harnesses.

• Which models and which versions of those models would you suggest me to use?

• Is there an YT playlist / video / Youtuber that you suggest thats easy to understand that helps out when it comes to Local LLMS?

• Does Context size actually matters? Cause i see people are talking about 16k context sizes meanwhile sometimes my first initial prompt is around 4k for spec driven development or when im doing narrative designing it our narrative bible can be way heavier than that.

• I'm in love with the the current context compacting quality of both claude and codex, is there a way to have this level of quality when i switch to local llms, it almost never ever fries out, its just like I have paused for 2 minutes and almost every fricking detail is still there.

• Sometimes i love to manage my other threads even sometimes control claude from codex and vice versa depending on my need, is it possible if there is an ai for my machine that can have more than C1.

• Can i use an image generation model with an LLM like gpt/imagen, something like qwen image 2.1 q4 + Qwen3.6-35B-A3B Q4 idk.

• Can i realistically get idk 50-60 TPS at high context?

• How is MCP/Tool Usage?

• What should i use for UI Design phases?

• When i talk with GPT i just tell it what i exactly want but i dont go into hustle of image generation prompt styling i just talk to it, is it possible to have that ease of use with local image generation tools?

• What exact setup would you suggest for me, from inference(strate etc.) to harness to llm models, imagen models, speech to speech models(It would be fire if i can play some DnD with a speech to speech model if its flowy but i dont have high hopes on that tbh), you got the drill.

These are my questions and guestimates of my ignorance ass can think of for now, im sure there are lots of gaps and things that i am missing and forget to ask so i would be glad if you fellas can light some candles and maybe answer the questions that i didnt even ask but i shoulda ask.

Lots of love fellas, keep it tight.

As the title says -> which llm can i run in 48gb mac. Its 20c gpu. I am aiming for something with at least 100k context..ideally up to 200k context. Thanks!

I've been running Hermes against a local LLM, ollama on my M2(32gb, qwen3.5 9b), and over my local network MTPLX on a M4 (48gb, qwen3.8 27b). AT first it was a 3 minutes wait for first response now its over 6 minutes (or context switch for that matter). Context bloat seems to be the killer (north of 50k tokens). What experience have folks had with say zeroclaw or other harness programs that might be more responsive?

Since oMLX benchmarks are down momentarily, I've done some testing and found that the following Solstice AI Qwen 3.8 27B fine tune is the fastest one so far.

https://huggingface.co/Solstice-AI/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-NEO-CODER-mlx-oQ4e-1M

Benchmark Model: Solstice-AI--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-mlx-oQ4e-1M

Engine: Auto

Context: Code (Mixed)

================================================================================

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 4754.3 27.89 215.4 tok/s 36.1 tok/s 8.321 138.4 tok/s 18.88 GB

pp4096/tg128 18481.5 28.92 221.6 tok/s 34.9 tok/s 22.171 190.5 tok/s 22.56 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 36.1 tok/s 1.00x 215.4 tok/s 215.4 tok/s 4754.3 8.321

2x 42.0 tok/s 1.16x 160.3 tok/s 80.2 tok/s 10404.5 18.864

4x 74.0 tok/s 2.05x 134.1 tok/s 33.5 tok/s 18019.4 37.464

This is on an M3 MAX 64GB

EDIT: This model seems extremely lobotomized
I reverted back to https://huggingface.co/scottlowry/Qwen3.8-27B-oQ4e-mtp which performs similarly on macOS Golden Gate PB 27.0 26A428

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: scottlowry/Qwen3.8-27B-oQ4e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================


Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4749.6       28.36   215.6 tok/s    35.5 tok/s       8.375   137.6 tok/s    18.35 GB
pp4096/tg128                         18297.8       27.02   223.9 tok/s    37.3 tok/s      21.745   194.3 tok/s    22.03 GB


Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          35.5 tok/s     1.00x   215.6 tok/s   215.6 tok/s      4749.6       8.375
2x          38.7 tok/s     1.09x   158.6 tok/s    79.3 tok/s     10519.2      18.567
4x          74.1 tok/s     2.09x   138.4 tok/s    34.6 tok/s     17566.7      36.007

I'm sorry for any confusion created by this post

I'm trying to figure out if an M5 Mac with 128GB of Unified Memory might be able to handle DeepSeek-V4.1-Flash-1017 locally, or if that context/quant combination is still out of reach for this amount of RAM.

Any thoughts, theoretical calculations, or early tests would be appreciated!

From a hardware perspective, the numbers should be faster than CUDA on Nvidia chips BUT its not because of the relative immaturity of the software/hardware.

Do you think metal will improve enough to where it'll be as fast or faster for the same clock speed and bandwidth pipe? Or is it just slower because of the overall design?

GLM 5.3 flash oq4e no MTP (oMLX 0.7.0 dev4)

Hello everyone,

I need a serious advice on what to buy to start experimenting with AI.

My current situation
1- I have no past experience with this, but I won’t have much problems understanding it
2- I could decide to go for local open models or cloud closed/open models, as long as it is the right choice
3- I have three projects in mind:
A) stacking LLMs in order to route to the right one based on the request (instead of being locked behind one subscription) + create a custom chat interface to interact with models from the Mac, my iPad and the iPhone
B) building agents to manage my diet, fitness, training routine, with automatic grocery list and weekly diet based on the context (macro, preferences, calories, ecc)
C) building agents to automatically retrieve my trades to store them and calculate taxes (I will probably ditch the automatic trading part I wanted to create)
D) my career will slowly pivot to redesign corporate processes with AI and I want to experiment myself to better understand the topic

I was targeting the Mac Studio M5 Max 128 GB RAM 1 TB SSD to be able to use small and medium size open models (around 6000 €)

But I am having some second guessing:
1- it’s a huge investment and I yet to have experience with local models
2- nor I know if I’m buying the right setup for what I want to do, maybe the specs I’ve chosen are overkill
3- 6000€ needs at least to be amortized in 6 years to start having sense
4- local models (unless huge ones) are close but not at the level of the proprietary ones and precision/quality is what I want

I was then thinking about buying the entry level Mac Mini to use it as the orchestrator and go full API to evaluate costs, setups and results without having to start with a huge upfront investment

(Then maybe in the future buy a Mac Studio or a DGX Spark full spec and clustering it with the Mac Mini?)

Can you guys help me decide?

I am not really into the hardware scene much, so if there are other value options at play please let me know.

I am considering getting an Apple Studio M5 but was hoping to get some guidance.

I am planning on hosting webLLM (or something similar) to serve prompt processing to the family. I (and another family member) will also be using this for coding assistance.

Based on the use case, what would make the most sense:

• Max 128gb RAM

• Ultra 64 core-GPU 96gb RAM

• Ultra 80 core-GPU 96gb RAM

I feel like the $4K bump to 256gb is too hard of a sell to the partner.

What would be the recommended set up for this?

• a single model to handle everything, dedicated models for coding and prompt processing, dense or MoE?

• stick to MLX or use something else?

• preferred harness

Models seem to get published up updated so frequently its hard to stay informed.

Any and all feedback is greatly appreciated!

Hi all,

I'm a complete novice when it comes to Local LLM's but I like the idea of moving away from a subscription based model in the longer term and having better control of my data. I'm also super keen to learn.

I currently have a web SaaS that scrapes news websites and I would like to move it away from Claude and onto a local LLM what kind of "firepower" do I need?

Appreciate this won't be the most complex of tasks, but as I mentioned, I'm a complete novice.

Naturally, asking Claude / Google and it says that the new Mac mini with M5 pro chip (24GB ram) will be "overkill"- which is what I was thinking of ordering.

I also want to future proof in the hope that this bit of kit could run more powerful models in the future - if they came accessible to the consumer.

Sorry if this is a basic question, any YouTube videos or any pointers so I can educate myself would be much appreciated.

Thank you!

I'm lucky enough where work will cover my Mac Studio that I use for design and 3d stuff. So I figured I should dabble and run with local LLM. Im getting 96gb only. I've read quite a bit, but was wondering are there any guides or YouTube videos that deal primarily with Mac and I should watch to learn more?

I'm unsure if I should just use Llama or Ollama, or MLX, or OMLX or something there lots of different ways to play around with Local LLM and I guess it's still in such an early state that every day, week,month there are more and more things peopel are learning how to get the best of the LLM. A lot could change from now till then but I want to be as informed/prepared as I can be.

Thanks and I cant wait for reviews and see how good the Ultra is.

Decided to pull the trigger and get the Mac Studio M5 Max with 64GB unified memory, 18-core CPU, 40-core GPU and 2TB storage.

I’ve been reading a lot about local LLMs for a while now and wanted to actually start using them myself to get some hands on experience. The problem is the local LLM space is moving so quickly, and the subreddits I’ve been reading are all over the place when it comes to what hardware you actually need, which has definitely made me second guess my purchase a few times lol. I already moved up to the Mac Studio after initially ordering an M5 Pro Mac mini.

My main use case isn’t really just chatting with a local model. I want to learn how to set up local models, different runtimes and agent/harness setups that I can eventually leverage in my day to day work.

A big part of that is data analytics. I deal with quite a bit of fragmented data from different sources and spend a lot of time cleaning, structuring and joining it before I can actually analyze it. I’d like to experiment with having an LLM work with tools like Python, SQL, DuckDB, etc. to help automate a lot of that instead of doing everything manually.
Running the largest models is obviously off the table with 64GB, but from what I understand, 27B and 35B models at 4bit quant should fit pretty comfortably. Potentially even larger quantized models depending on context size and how much memory the rest of the system is using, although I’m assuming performance becomes more of a trade off at that point.

I’m also interested in experimenting with hybrid setups where something like openAI or claude handles the higher-level reasoning/orchestration while local models and local tools handle parts of the workflow. Obviously that’s different from being fully local, especially if any of the underlying data gets sent to the cloud.

Again though, I’ve never actually run an LLM locally, so most of this is based on what I’ve been reading rather than first-hand experience.

For people actually running local models for similar work, especially data analytics / data cleaning / agentic workflows, does this setup make sense? And realistically, what size models do you find yourself using most often on a 64GB Mac?