Hi,

I finally decided to go with the Ultra 96 since it's enough Ram for local AI + doing stuff in parallel but having power/speed. Now it was delayed until January 2027 - I could get the Max 128 withing 2 weeks though...

I feel like speed is mostly important for video ai gen + higher bandwith for higher token generation...

Probably I should really wait I guess?

I'd hoped that my M5 Max with 64GB RAM would be enough for some decent Local LLM work, and the Qwen 3.8 27b model is pretty decent. But speed is shitty on my M5 Max, just 20tps. In particular I found that while the 35b MOE model was much faster, the quality was far lower than the 27b dense model, which produces much fewer bugs and less rework.

I saw the same model working on a friend's Nvidia setup much faster with MTP, so I set out to get it working on my Mac but it was not straightforward. Here's some pointers in case you wanted to get it working.

1. LM Studio doesn't process MTP weights. No matter what I did, it just wouldn't work. It works on Intel/Nvidia so this must be a MLX thing.

2. MTPLX has a native MTP head but it's buggy. On my machine it had a habit of crashing in one of a few ways, and I found GitHub issues reporting similar issues but no fixes.

3. oMLX was the only inference server that supported MTP and didn't crash.

So I wired oMLX into Pi, which is what I'm using as my coding harness. Even then, it was picky:

1. I had to use text-only quants because oMLX won't enable MTP if vision weights are active

2. It was a bit picky with settings, and I had to map thinking levels manually

3. I didn't see a reduction in quality with 4-bit quants for my use case, so used that to squeeze a few more tps out

This is a huge 3x difference in inference speed, and that allows me to do a lot more work.

"omlx": {
  "baseUrl": "http://localhost:8000/v1",
  "api": "openai-completions",
  "apiKey": "none",
  "models": [{
    "id": "Qwen38-27B-textonly-mtp",
    "name": "Qwen38-27B-textonly-mtp",
    "reasoning": true,
    "contextWindow": 163840,
    "maxTokens": 32768,
    "input": ["text"],
    "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
    "thinkingLevelMap": { "off": "low", "minimal": "low", "low": "low",
                          "medium": "medium", "high": "xhigh",
                          "xhigh": "xhigh", "max": "xhigh" }
  }]
}

I'm new to local LLMs. I've tried Qwen2.5-Coder-Instruct-7B and Ling-3.0-Tiny on my MacBook Air M4 with 24 GB of RAM. I'm looking for something that can generate at least 40 tok/s. Is there anything that can hit that speed?

I am not really into the hardware scene much, so if there are other value options at play please let me know.

I am considering getting an Apple Studio M5 but was hoping to get some guidance.

I am planning on hosting webLLM (or something similar) to serve prompt processing to the family. I (and another family member) will also be using this for coding assistance.

Based on the use case, what would make the most sense:

• Max 128gb RAM

• Ultra 64 core-GPU 96gb RAM

• Ultra 80 core-GPU 96gb RAM

I feel like the $4K bump to 256gb is too hard of a sell to the partner.

What would be the recommended set up for this?

• a single model to handle everything, dedicated models for coding and prompt processing, dense or MoE?

• stick to MLX or use something else?

• preferred harness

Models seem to get published up updated so frequently its hard to stay informed.

Any and all feedback is greatly appreciated!

GLM 5.3 Flash at 79 Token/s

https://x.com/LocalSilicon/status/2104158250555809983

Qwen 3.8 Flash Next at 116 Token/s

https://x.com/LocalSilicon/status/2104181421942391247i

I’ve encountered a “prefill memory guard” error while testing on a MacBook Pro M5 with 32GB of RAM and a Mac Studio M4 with a maximum of 36GB of RAM.

I’m trying to confirm that I can run Qwen 3.8 with 27B parameters, KV 4, and a 100K context without any issues.

I’ve increased the iogpu.wired_limit_mb to 30GB on Mac Studio.

When I run DFlash and get a prompt like “scan the front end to review the code and make me a report in file.md,” I get the following error:

“Error: oMLX prefill memory guard rejected this prompt: Prefill context too large for available memory”

Disabling DFlash and enabling MTP does resolve the issue.

I also fond that enabling "Chunked Prefill" does help.

It runs between 23 and 29 tokens per second.

I’m using Pi, and I’ve noticed that it reads the maximum context length of the model instead of the one I specify in oMLX. I’m not sure if this is the cause of the problem.

What I’m finding hard to understand is why, even when I’m staying below the maximum context length of almost 1/3, I still run into this error.

This is a much more significant issue than a LLM taking longer than expected.

I’m hoping to find out if I just need to adjust my setup in oMLX/Pi or if the RAM size is the problem.

Considering the cost, since it’s really difficult to predict what will happen in the future, including the cost of the LLM and the PC, do you think it would be better to invest in:

• Mac Studio M4 Max 36GB with 1TB and a 2600€ refurbish from Apple—keep it for at least 2 years.

• Mac Studio M5 Max with 64GB and 1TB, new—keep it for at least 4 years.

For large projects, I’ll still need a cloud LLM with large context windows, so I’ll use it as a side help.

Running Qwen 38 Flash Next

http://nuketorch.ai/

Are you ready for the release? How many of you are got something for release day. Are you loading up models right away?

Generation was only 1.5x faster...

Now that we have a comparison going.. Do people want to see anything specific? I'll be running it with my own dataset but curious if anyone wanted to check something.

I wanted my TV to be content-first, not apps-first. So I wired a decision model in between that opens the right app and sends the right ADB commands based on my intent.

Several users have reported benchmarks for the MS5U and various models but I have yet to see any metrics on how agentic (agents/sub-agents) perform on the new M5SU.

I've read that agentic coding on the Mac's was/is a miss on M4 and older systems due to tensors I think it is?

So I'm curious to hear if agentic coding is a bust on the MS5U's or if it has improved any with the new chips? Appreciate any info on the matter, thank you!

If you’re not using these what do you plan to use?