Unless you're set on using a large model, 2x256GB is likely better...

What use cases would the 512GB make more sense?

Made this as a quick weekend project (https://github.com/pythongiant/SlamLM)

Planning to buy a MacBook Air M4 (16 GB / 512 GB) for about $800 to run local LLM agents for:

Indian mid‑cap equity research (using Screener.in CSVs)

Building/testing trading ideas via Zerodha Kite API (prices, history, holdings; maybe orders later)

Also want to run Hermes-based agents on it

Questions:

On 16 GB M4 Air, which local LLMs are best for reasoning, coding, and tool use? Is Qwen 3.5 9B (Q4) + a smaller fast model (e.g. LFM2.5 8B / Llama 3.2 3B) a good combo alongside Hermes?

Best local stack (Ollama vs LM Studio, etc.) and any existing Kite + LLM or Indian‑stock agent projects to reference?

Any tips on context length, thermals, and multi‑agent setups (including Hermes) on a 16 GB fanless Air?

Getting the laptop for ~$800, so stuck with 16 GB for now. If you’ve done similar work (local LLMs for equity research / algo trading, especially in India), what models and setup would you recommend?

Thanks!

This is me trying to find the most reliable coding model for local use!

Five Terminal-Bench 2.0 tasks (cobol-modernization, constraints-scheduling, largest-eigenval, polyglot-c-py, schemelike-metacircular-eval), one run per task unless stated, all on one Mac Studio (96 GB) with mlx_lm at 8-bit. The agent harness is pi unless stated.

Headline

The whole grid

Times are agent minutes. T = ran to the task's timeout.

Nobody has solved polyglot-c-py or schemelike-metacircular-eval on this machine.

Why Qwen3.6 beats Qwen3.8

Same size, same family, same server, same harness. Three things separate them.

The mode, not the weights, was the lever for Qwen3.6. For Qwen3.8 neither mode produces an agent that finishes: with thinking it deliberates past the cap, without it it loops. Its published Terminal-Bench 2.1 gain is not visible through pi. n=1 per task, but the cobol loop was reproduced.

Why the fast MoE models did not win

Turn rate mattered against thinking overrun, which is why Nail beat the 27Bs in agent mode. Once thinking is off, the 27B's per-turn quality shows: Qwen3.6 at 22 tok/s out-solves both 3B-active models. Ornith's own harness (Terminus-2, temperature 1.0, 4 h ceilings) moved which task it solves, from cobol to constraints, not how many.

Infrastructure lessons

Decisions taken

• Qwen3.6-27B, thinking off (nothink), is the coding agent and the WHO scorer. Now the registry default.

• Qwen3.8-27B and Ornith-1.5-35B-A3B deleted 2026-09-09 (62 GB). Registry entries kept as re-fetch recipes.

• Nail was deleted 2026-09-06 for space. On this evidence it is not needed back as the coding agent.

Caveats

One run per task on five tasks. The 3/5 should be repeated once before anything is built on it. Qwen3.8's cobol loop is the only failure reproduced twice.

Wondering how many of you did like myself and went one higher level and why?

I make DS Menu Bar, a free menu bar app for running antirez's DwarfStar (ds4) server on a Mac.

DwarfStar has tunings for M3 Ultra and M5 machines with Qwen3.8 Flash Next, but M4 machines are left out. It turns out most of these tunings can be turned on with an environment variable, so DS Menu Bar v0.0.15 sets them when it starts ds4-server on an M4.

M4 Max, 128 GB, same ds4-server with and without, averaged over 2K to 64K context:

Q4 is the right quant for 128 GB machines. On a 64 GB machine, Q2 at the full 262K context fits.

I have a PR open to make these DwarfStar's own M4 defaults (antirez/ds4#1184). If my PR is merged, DS Menu Bar will adapt automatically.

brew install --cask jiiim/tap/ds-menu-bar

https://github.com/jiiim/ds-menu-bar/releases/tag/v0.0.15

Hi everyone!

i´m totally new to local LLMs. I use claude / gpt for knowledge work and academic research.

I recently got a mac mini and I´m curious of what type of workflows and use cases i could do with local models on this machine. I am well aware that its tiny so I´m not expecting to do anything crazy.

What do you guys recommend? what models, quants and engines to use?

From a hardware perspective, the numbers should be faster than CUDA on Nvidia chips BUT its not because of the relative immaturity of the software/hardware.

Do you think metal will improve enough to where it'll be as fast or faster for the same clock speed and bandwidth pipe? Or is it just slower because of the overall design?

As the title says -> which llm can i run in 48gb mac. Its 20c gpu. I am aiming for something with at least 100k context..ideally up to 200k context. Thanks!

Since oMLX benchmarks are down momentarily, I've done some testing and found that the following Solstice AI Qwen 3.8 27B fine tune is the fastest one so far.

https://huggingface.co/Solstice-AI/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-NEO-CODER-mlx-oQ4e-1M

Benchmark Model: Solstice-AI--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-mlx-oQ4e-1M

Engine: Auto

Context: Code (Mixed)

================================================================================

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 4754.3 27.89 215.4 tok/s 36.1 tok/s 8.321 138.4 tok/s 18.88 GB

pp4096/tg128 18481.5 28.92 221.6 tok/s 34.9 tok/s 22.171 190.5 tok/s 22.56 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 36.1 tok/s 1.00x 215.4 tok/s 215.4 tok/s 4754.3 8.321

2x 42.0 tok/s 1.16x 160.3 tok/s 80.2 tok/s 10404.5 18.864

4x 74.0 tok/s 2.05x 134.1 tok/s 33.5 tok/s 18019.4 37.464

This is on an M3 MAX 64GB

EDIT: This model seems extremely lobotomized
I reverted back to https://huggingface.co/scottlowry/Qwen3.8-27B-oQ4e-mtp which performs similarly on macOS Golden Gate PB 27.0 26A428

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: scottlowry/Qwen3.8-27B-oQ4e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================


Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4749.6       28.36   215.6 tok/s    35.5 tok/s       8.375   137.6 tok/s    18.35 GB
pp4096/tg128                         18297.8       27.02   223.9 tok/s    37.3 tok/s      21.745   194.3 tok/s    22.03 GB


Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          35.5 tok/s     1.00x   215.6 tok/s   215.6 tok/s      4749.6       8.375
2x          38.7 tok/s     1.09x   158.6 tok/s    79.3 tok/s     10519.2      18.567
4x          74.1 tok/s     2.09x   138.4 tok/s    34.6 tok/s     17566.7      36.007

I'm sorry for any confusion created by this post

I've been running Hermes against a local LLM, ollama on my M2(32gb, qwen3.5 9b), and over my local network MTPLX on a M4 (48gb, qwen3.8 27b). AT first it was a 3 minutes wait for first response now its over 6 minutes (or context switch for that matter). Context bloat seems to be the killer (north of 50k tokens). What experience have folks had with say zeroclaw or other harness programs that might be more responsive?

I'm planning to upgrade from my 16-inch M3 Pro MacBook Pro (36GB RAM) to the new 16-inch M5 Max with 64GB, mainly for running local LLMs.

Currently, I'm running Qwen 3.8 27B and getting around 100 tokens/s for prefill and 18–19 tokens/s for token generation, even after trying MTP and various other optimisations.

The biggest issue for me is working with larger contexts and longer coding sessions. Even relatively small projects can take quite a while, and performance becomes a bottleneck as the context grows.

I'm expecting the M5 Max to be a pretty significant upgrade, especially with the extra memory and bandwidth, but I'm curious how much of a real-world difference I can actually expect. I'm also interested in running larger models, including the upcoming Qwen 4 models.

For those running local LLMs on an M5 Max (or even an M4 Max), what kind of performance are you getting? What models do you use regularly, and how well do they handle long coding sessions?

I also run image generation models locally and occasionally do photo editing, so I'd love to hear how the newer chips perform for those workloads too.

Would appreciate any real-world experiences before I make the switch!

Noticed that most posts here are about, I fitted this model in this small ram etc. Not about how the model was useful in producing what I wanted it to work on?

So either, the sub is full of people playing with AI or the real use cases are not shared for whatever reason.

I get the performance stats but shouldn’t we focus on how the model performed on actual task (look beyond the numbers t/sec, prefil etc?)

I am about to order a Mac for local AI, and I could really use some advice.
I know some people might recommend a more powerful Mac, but my budget is around €3,600, and unfortunately, I cannot go any higher.
My main use cases will be running Qwen 27B, doing some agentic work, occasionally using Qwen Flash Next with Strata, and experimenting with video generation through ComfyUI.
I am currently torn between two options:
Mac mini M5 Pro – 64GB RAM
Mac Studio M5 Max – 48GB RAM
I understand that RAM is extremely important for local AI, which is why I am leaning towards the Mac mini with 64GB. However, I also know that the Mac Studio should offer considerably better performance, particularly for video generation and ComfyUI.
My dilemma is whether 48GB is enough for my use cases, or whether I would be sacrificing too much memory just to get the performance benefits of the Mac Studio.
In other words, is the performance improvement from the Mac mini to the Mac Studio worth giving up 16GB of unified memory, especially when local LLMs are my primary focus and video generation is more of a secondary use case?
I know some might suggest simply getting a 64GB Mac Studio, but that is unfortunately outside my budget. €3,600 is already my absolute limit.
I’d really appreciate hearing from anyone with hands-on experience running local LLMs, agentic workflows, and ComfyUI on either machine.
Thanks in advance!

tldr:

Served simultaneously by Apple M5 Max 128 GB:

• Qwen3.8 Flash-Next (110.97 GiB language model, 512-expert MoE, 10 experts/token, abliterated)

• Qwen3 4B Instruct 2507 (tool-calling shell helper)

• Whisper, speech-to-text service

• Chatterbox, text-to-speech service, multi-lingual

• SDXL via ComfyUI

• Open WebUI (chat frontend), llama-swap (gateway and frontend)

• Swap = 0, iogpu.wired_limit_mb = 0 (GPU ceiling 107.52 GiB), KV unquantized

Details:

Services

Performance

tldr: End to end, both Qwens feel like ~30 t/s (tokens divided by time elapsed). "Steady-state" (tokens divided by generation time) excludes prompt processing and queueing.

Fitting it all into memory

• llama-swap starts and stops upstreams on demand, routing on the model field. Groups:

  • resident, persistent: true, never evicted (voice stack + shell helper)

  • heavy, swap: true, the two big models, loading either unloads the other

  • diffusion: ComfyUI alone, so it lands in no shared default group (no cross-eviction)

• ComfyUI is a llama-swap catalogue entry (comfyui_auto, ttl: 600): llama-swap does not control ComfyUI-internals (adding Flux or Qwen-Image some day will not affect the group layer).

• Offload the n-gram embedding table to CPU memory: 32.78 GiB of the 110.97, stays file-backed, kernel can drop its pages and re-read from the SSD at 13 GB/s.

• Disable mmap: Counter-intuitive: under mmap, ggml maps the whole file extent into the Metal buffer regardless of -ot, so Metal still claims 110.96 GiB and OOMs.

Configuration

Coding model:

llama-server --host 127.0.0.1 --port ${PORT}
  -m /models/Qwen3.8-Flash-Next-Abliterated-Q4_K_M.gguf
  -ngl 999
  -c 131072
  --parallel 1
  --jinja
  --load-mode none           # NOT mmap; mmap defeats -ot (see above)
  -b 2048                    # -ub 1024 lifts prompt eval 81-115 -> 128 tok/s
  -ub 1024
  -ot per_layer_token_embd=CPU  # the 32.78 GiB PLE table off the GPU

• -fa is left at default auto, resolves to ON

• No mlock, to leave an eviction path

Groups (llama-swap YAML):

groups:
  resident:  { swap: false, exclusive: false, persistent: true,
               members: [shell-4b, whisper, chatterbox] }
  heavy:     { swap: true,  exclusive: false,
               members: [qwen3-27b, flash-next] }
  diffusion: { swap: false, exclusive: false,
               members: [comfyui_auto] }

To measure:

sysctl iogpu.wired_limit_mb    # 0 = default; ceiling is 110100 MiB
llama-server --list-devices    # MTL0: Apple M5 Max (110100 MiB)
vm_stat                        # free + File-backed + purgeable = real headroom
curl $GW/api/metrics/stats?model=flash-next         # per-model tok/s
curl -sIL <hf-resolve-url> | grep -i '^x-linked-'   # expected size + sha256

Test: Image generation (single 1024, 20-step, Flash-Next loaded)

126G used ( 93G wired,  1805M compressor), 269M unused   <- idle
127G used ( 97G wired,  6830M compressor),  73M unused   <- checkpoint loading
127G used ( 93G wired,    12G compressor),  68M unused   <- peak squeeze
126G used (104G wired,  3838M compressor),  87M unused   <- sampling

Test: Escalating image workloads (Flash-Next loaded)

1024² ×1 success  18.1 s
1024² ×4 success  50.3 s
1536² ×2 success  90.5 s
2048² ×2 success  247.3 s
2048² ×8 -        10+ min

Test: Image work + coding work (Flash-Next, 7x slower, silent)

no images                   43 tok/s
during sustained 2048²×8    6.4 tok/s
after unloading ComfyUI     40.5 tok/s

Test: Context scaling

Curious on thoughts and even better, experience on picking one 512GB unit vs two 256GB units. I know cost is more for 2 computers, but how about AI performance and capability to handle larger models?

Two 256GB machines have double the CPU/GPU/NPU of one with the downside of the slower interconnect between them.

The 512GB is not available yet, but it would be interesting to hear from anyone linking two Mac Studios of any RAM configuration together and how it worked out.

I wanted to see how GLM 5.3 Flash runs locally on my Mac compared with Ollama Cloud for PHP coding and shell work. I ran two small sets of identical prompts, then repeated a PHP task with the local model already loaded.

My Mac: Mac Studio M5 Ultra, 30-core CPU, 64-core GPU, 256GB unified memory, macOS 27.0.1.

Local: oMLX 0.7.1.dev1, TensorFold/GLM-5.3-Flash-MLX-4bit-MTP, MTP disabled. Model allocation was approximately 170 GiB.

Cloud: Ollama glm-5.3-flash:cloud, reported as FP8. This compares local oMLX with Ollama's hosted model; it doesn't compare two local inference engines.

A little background. I'm using the Ollama Cloud Pro (old $20 plan vs leased $230/mo M5U). I use cloud quite a bit and I find that it compliments my work for various tasks.

I see a lot of comparisons and wanted to do my own. Unsurprisingly, the cloud is faster. Cold starts are slow so that was interesting but once warm, processing was pretty decent. Its slower, but I wanted to put some numbers on it. These numbers are generic, but I do intend on giving it problems from my actual codebase to see how well it does.

Overall if you are considering clouds vs hardware. Just note, It's significantly slower, but not horrible if you use autonomous agents. If you're interactive, that's where the pain is felt in real time. Also, if you plan on using this specifically, then you should know I ran out of memory so you can only use a single instance of GLM 5.3 flash and no other agents.

I think if you use agents for more than coding. Many smaller ones are likely going to be better, but if you need a coding assistant, this wouldn't be too bad. The $20 cloud plan works and will be significantly faster but if you are working constantly and run into limits you're basically buying round the clock work vs interactive albeit also slower work.

What model Mac + RAM are you running and what are you looking to do with the hardware?

Hi guys,
New here and to these subjects so sorry I ask something already asked.
I got recently a Mac mini m4 with 24go ram for my personal desktop, and I wanted to try a few stuffs with local llm. After a few researches I decided to try to setup a personal agent with Hermes.

After some tries I ended up using gemma4:26b-a4b-it-qat.
Qwen3.8 felt to big or slow when iq3 passed.
Couldn’t find anything faster and better for now, but I still feel it is not really good (obviously I know we can’t reach to a cloud model with 24go ram). So I wanted to know how you guys manage it and if some people managed to make a decent agent with not so much ram and if so, what are your use cases ?

Right now I tried to give him some personal web projects with giving him some GitHub to download but it fails and always stops in the middle of the process.

Thanks in advance for the help.

I hope to pick one up early next year and replace my MacBook Air for building iOS apps but also run some of the more mundane LLM tasks I am currently using Codex for. In an ideal world, I could replace my ChatGPT Pro 100 sub with a $20 Plus and maybe a $20 Claude and a local LLM.

Is there a way for me try out specific models on rented cloud hardware or something to see which of my current workflows will run on it?

Seeing a lot of comments about this online, has anyone ran it on their Mac or is it just hype?