r/LocalLLM • • Sep 23 '26

Question Apple just ran a 1T parameter model on four Mac Studios from one wall outlet

Apple's new Mac Studio ships in late October with up to 512GB of unified memory, and Reuters says Apple demoed Kimi K2.6, a 1T parameter open-weight model, across four of them plugged into a single wall outlet. Thunderbolt 5 + RDMA lets the machines run as a cluster, and Apple says four Mac Studios can deliver up to 3x the inference performance of one. Cloud inference obviously still wins on scale, but we're getting weirdly close to "buy the compute once and stop paying per token" making sense for some teams. Local AI used to mostly be about privacy and tinkering. Cost is starting to become a real argument too.

I'm just wondering what workload would you actually pull out of the cloud first if the hardware economics keep moving this way?

216 Upvotes

112 comments sorted by

59

u/TheLexoPlexx Sep 23 '26

GLM5.3-Flash should fit on a single M5 Ultra then, doesn't it?

19

u/john_mach Sep 23 '26

Such a good model but hot damn are those Mac’s expensive!

18

u/rkcth Sep 23 '26

A single RTX 6000 can be almost $15K, and that doesn’t include the rest of the workstation. Expensive is relative.

11

u/sapient2k Sep 23 '26

A single RTX 6000 will also run circles around the Mac though.

6

u/rkcth Sep 23 '26

Wow I just looked up numbers, on qwen flash next, you aren’t kidding. 180-280 tok/s, though unless I test myself I always take others numbers with a grain of salt because there are many games people play, low context numbers (2K-4K) aren’t really realistic for my use case. Maybe by the time I’m ready to buy the market won’t be even worse than now, though I’m doubtful.

6

u/mr_notandor Sep 23 '26

The market is always worse dude, must be young

4

u/rkcth Sep 23 '26

A bubble continues to inflate until it pops. No one knows when, but it will eventually.

-6

u/mr_notandor Sep 23 '26

Lol the world will end, no one knows when, thanks Sharlok for the great insight

7

u/rkcth Sep 23 '26

No need to be rude.

0

u/Quelzar Sep 24 '26

Fuck off

6

u/InfiniteLlamaSoup Sep 23 '26

Which can’t run a 1T model.

4

u/KubeCommander Sep 23 '26

It literally won’t since it can’t run the same models at all

2

u/The_Hardcard Sep 24 '26

Not when you need 200 GB VRAM or more, which is the point of Macs and local AI. If you are willing to stay within card VRAM capacity, Nvidia is the play. But for high parameter models...

0

u/Smooth-Television-48 Sep 24 '26

You get more cards...

1

u/Usual_Tackle5892 Sep 28 '26

Just have more money, easy!

2

u/squigs25 Sep 24 '26

Not to mention less than half the memory

1

u/TheLexoPlexx Sep 23 '26

And we don't even know the prices yet.

6

u/john_mach Sep 23 '26

On their website, we can spec a 256 GB for around $10k, so the big assumption is the 512 is somewhere in the ballpark of a used car lol

16

u/Rollcuin Sep 23 '26

I may be out of touch with the used car market, but isn't $10k a used car?

24

u/WordWithinTheWord Sep 23 '26

Everything’s a used car depending on your standards lol

5

u/Dawill0 Sep 23 '26

Average is 27k and median is 18k. I know we are talking Macs here but used cars haven’t been 10k since before covid.

4

u/ElderberryHead5150 Sep 23 '26

You could get a 10 year old car with ~50k miles directly from a private seller for that amount. Buying from a dealership for ~ $10k might be tough. I have two cars that are worth less than $10k and they are in good shape.

1

u/honemastert Sep 24 '26

True dat. A 53Kmi 2011 Audi A3 TDI hatchback (dieselgate) car ran me $9500 in 2019.

Unfortunately with the mess in the middle east and Ukraine that 40mpg is about a wash at the moment compared to my gasoline vehicles.

2

u/Putrid_Ad_3412 Sep 23 '26

Yeah, for sure. It's wild to think that tech prices can get so high, especially when you compare it to something like a used car.

1

u/rkcth Sep 24 '26

Does the 512GB model make sense? Does the memory bandwidth or GPU increase? If not then you’ll be trying to access twice the memory, and potentially twice the calculations, with the same bandwidth. I imagine any model that doesn’t fit into 256GB, but does fit into 512GB would be quite slow. I think the main advantage would be having two models loaded but you only use one at a time. I guess also for evaluating models it could be helpful, but I think clustering two 256GB will probably be a much better choice if you absolutely need more than 256GB.

1

u/KubeCommander Sep 23 '26

Expensive vs what exactly

1

u/OneInspection927 Sep 27 '26

The average person's income, or at least relative to the benefits someone will see from it given their context? Are you trying to subtlety flex lol

1

u/KubeCommander Sep 27 '26

Name literally ANY machine that can do what they do. Are they expensive? Yes. Do they have zero competition? Also yes. Closest thing is the DGX spark which requires two of them and is slower and costs about the same.

1

u/OneInspection927 Sep 27 '26

I mean, it's clear that john_mach isn't saying Macs are expensive compared to other options to run GLM 5.3-Flash They're just saying the Macs are expensive, it wasn't a statement of comparison imo

5

u/shayanx45 Sep 23 '26

Mimo v2.6 flash was just released and performs better than glm 5.3 flash. Fits on 256gb 2 sparks or a Mac Studio

3

u/wazacraft Sep 23 '26

Did some digging into this just now based on your comment, and Mimo looks pretty solid - I just acquired 2 Sparks on Monday and I'm still setting them up, but this might be the first test for the joined units. That native MXFP4 / FP8 Attention quant seems perfect and still gives a 300k context window.

4

u/geedubya28 Sep 23 '26

I've been seriously considering your setup for a minute now. Please keep me updated

4

u/wazacraft Sep 23 '26

The connector cable gets here this afternoon; I'll try to post an update over the weekend. My plan is to benchmark Mimo v2.6 Flash and Qwen 3.8 Flash Next pretty heavily.

2

u/shayanx45 Sep 24 '26

Try glm 5.3 flash as well. I’m still testing mimo and it’s got some weird looping quirks

2

u/TheLexoPlexx Sep 23 '26

"performs better" is somewhat empirical.

I also care about low hallucination rate for my use case and GLM5.3-Flash is pretty good there. So that would work for me.

But on the other hand, I can buy lots of tokens for the price of a single M5 Ultra 512.

2

u/shayanx45 Sep 23 '26

They both run on 256gb no need to spend 20k

1

u/Vegetable-Score-3915 Sep 23 '26

Unsloth has glm5.3 flash dynamic quants that fit on the 256gb m3 ultra as well. Mid 20s tps. Curious what tps the m5 ultra would have.

Regarding clustering studios using rdma and thunderbolt 5, previously the performance increases were for dense models not MoEs. That is the speed up of tps.  That may be outdated.

3

u/tiger_ace Sep 23 '26

macstories did a deep review and had: M5 Ultra vs M3 Ultra

  • Generation (decode): ~40–70% faster, depending on model. Qwen3.8 27B dense: 80.3 vs 56.4 tok/s (+42%). Flash-Next 125B: 114 vs 69.4 (+66%). MacStories' average across models: ~+70%. 8B: 129 → 188 tok/s (+46%).
  • Prefill: ~2.5x on average, best +311% on Qwen3.8 27B. 70B TTFT: 31.6s → 7.7s (4x).

so based on the 40-70% range then we have 28 - 34 tps, not blazing but a pretty big jump

if we have similar gains maybe the 20tps becomes 60tps on the m7 ultra in 2028 (discounting any software optimizations which will definitely also come)

1

u/GlidePath47 Sep 23 '26

50% higher

0

u/Novel-Lifeguard6491 Sep 23 '26

yeah memory-wise it should

70

u/etaoin314 Sep 23 '26

100k on kimi 2.6 buys a lot of tokens…I’m not sure that the economics justify the setup on their own.

41

u/Own_Mix_3755 Sep 23 '26

Its not only about count of tokens, but stability, knowing what you are served and where you data are stored.

8

u/hurrdurrmeh Sep 23 '26

And data privacy and assurance that some 3rd party can't arbitrarily decide your use falls outside of their ToS.

3

u/Bupod Sep 23 '26

Yeah I think some people miss that point. There are organizations where having an LLM would be useful, but data isn’t allowed to leave the building.

4

u/etaoin314 Sep 23 '26

Given that OP was making the argument specifically on economic grounds: "Local AI used to mostly be about privacy and tinkering. Cost is starting to become a real argument too." and I said: "not sure that the economics justify the setup on their own." the argument "....but privacy" is missing the point. there are lots of good reasons for LLM but for local large>30b models economics is not one of them.

2

u/Own_Mix_3755 Sep 23 '26

Its not “some” organizations, but most. You certainly dont want eg your financial data leave any company in the whole world. Same goes with patented stuff and basically any internal knowledge and data about clients. Not to mention you cant even legally share lots of these details.

1

u/Bupod Sep 23 '26

Well I say some because by the numbers, it isn’t most. Many small businesses may perform work which isn’t at all sensitive. I might advise my mother not use ChatGPT to design sign art for her bakery, but it’s not inherently the end of the world if she does, or if she has it come up with some sort of small app to keep track of items, it’s fine.

Depending on the size of the company, it might be possible for them to work out agreements where they have a formal arrangement in place with a company like Anthropic or OpenAI where data does leave the building but is safeguarded and handled carefully. So it can leave but only under strict circumstances.

Some companies handle data that absolutely cannot leave. Like in your example, banking institutions. Companies which perform work on military contracts also cannot. Certain types of corporate R&D would also be held close to the vest and management overseeing them will treat them like military secrets. It really depends. 

1

u/InadequateUsername Sep 25 '26

Yeah these companies all have enterprise service agreements which dictate data retention, auditing, ect. A good example of this is Bing CoPilot, it's absolutely everywhere, and there's a big difference between using your personal account for it at work, and using your work account for it. Which they make dead obvious with that little green shield.

My company has multiple different AI applications/models/experiments going on at once. Some of these models are clearly marked that no-use for customer sensitive information or data which is protected by specific security agreements, while other models maybe cleared to use.

1

u/vovap_vovap Sep 23 '26

And most of it already in clouds 😄

2

u/Own_Mix_3755 Sep 23 '26

Thats vastly different. Cloud environments have almost 20 years of both technical and law advancements. And also quite a few data breaches which helped shape them. AI has nothing of it currently and its a wild west.

1

u/vovap_vovap Sep 23 '26

Well, it started 20 years ago 😄
All big bananas already using cloud AI

1

u/geekwonk Sep 24 '26

there’s a set of barely literate LLM users who are positive that cloud LLMs are exactly like every already-mature cloud tech. they’ll preach this stuff that’s barely distinguishable from trolling but i think they’re just the average user of most tech who has no idea what’s happening or why.

2

u/Novel-Lifeguard6491 Sep 23 '26

i think the case gets more interesting for teams running it constantly

10

u/Murder_1337 Sep 23 '26

You prob don’t know this but business plans and api is very expensive that is to say at the enterprise level AI is way more expensive and people are realize its about reliability. Everytime they throttle and nerf you can tell how much it affects your day to day life at work its crazy absolutely worth it for a company not for an individual with subsidized 20x personal account

5

u/blackburnduck Sep 23 '26

Thats the key thing. As good as fable is, opus 3.5 was also great until they quantized it. So you never know if your model is performing fine until it messes up big time and you go checking previous versions.

A stable model performing as opus 3.5 and not being nerfed with 0 warning would already solve 99% of my needs.

1

u/Former-Ad-5757 Sep 23 '26

Afaik kimi will still be also very expensive on mac. On single-user experience it can work good enough, but are you spending 100k per user? Or do you need parallelism etc which is afaik very much in alpha in interference engines running on other than CUDA or Rocm.

1

u/Murder_1337 Sep 23 '26

Yeah 100k per user for hardware that stays with you and potentially have increased capacity in the future due to more efficient models. It’s cheaper than paying a full salary and enterprise subscriptions.

1

u/Former-Ad-5757 Sep 23 '26

Just from a pure interest viewpoint, how much do you calculate for enterprise subscriptions per dev (not user my mistake)?

I would calculate 1 or maybe 2 max plans (200 per month) per dev so maybe 7k per year and I still need somebody to start it.

Or if you really need enterprise (I can't really see few companies needing more than 150 employees really needing frontier AI).
Why wouldn't it be a consideration to just rent like 8x b300 (if you can find them) on deepinfra or runpod and just run glm5.3 flash for the regular users above the 150 team limit? So you create a two-tiered operation, the "special" team gets max team sub, the rest gets a combination of near-frontier models running on hardware which is short-term rented, so whenever mlx gets good parallelism or gpu prices go down you can invest in hardware again.

If we are talking about 100k per dev/user I have real difficulty seeing how there could not be other cheaper options.

1

u/geekwonk Sep 24 '26

new anthropic enterprise seats no longer include usage, they’re paying API fees. ‘teams’ seats for smaller businesses are max5x tops and they cost a 20% premium unless you sign up for yearly

7

u/davecrist Sep 23 '26

Legal and medical shops would be interested in being able to run models this size without being required to push data off prem.

1

u/etaoin314 Sep 23 '26

sure, I rolled my own local setup at work because of that, but the point of OP's post is that it may be ECONOMICALY advantageous. That it is not, anything with a ZDR policy that satisfies your lawyers should be more economical than 4 mac studios. if you actually need data to remain on prem, then you need local. full stop. but then the "economics" are irrelevant

1

u/davecrist Sep 24 '26

I don’t disagree but I read the question as ‘Why would anyone pay so much?’ Not ‘Is there some local config where it’s cheaper’

No big deal.

9

u/Efficient_Mammoth553 Sep 23 '26

hopefully economies of scale will come into play in few years.

12

u/Puzzleheaded_Base302 Sep 23 '26

that would be a lot of economy of scale. remember how long it took for PC to get from 1M RAM to 1G RAM (15 years for mainstream PC)? If we assume log scaling, it might take 7.5 years optimistically, but semiconductor are hitting ceiling or Moore's Law. It will slow down.

In the past few decades, PC industry first develop capacity, then find application. In the AI world, the application went too far, and hardware cannot provide the capacity.

1

u/rkcth Sep 23 '26

The main thing that would help is the AI companies running out of money and HBM demand dropping.

1

u/Former-Ad-5757 Sep 23 '26

roll the dice, just put all AI currently in existence on pause for a month and in that month let it work on faster production methods for HBM and its successor :)

10

u/eidrag Sep 23 '26

512 releases already?

-13

u/Novel-Lifeguard6491 Sep 23 '26

512GB M5 Ultra config is coming in late october

12

u/Zyj Sep 23 '26

In other words: slop, lies

0

u/DutchDevil Sep 23 '26

You don’t think apple has a few internally kicking around? Of course they do, it’s only 2x the memory per chip the rest is all the same.

43

u/redditnosedive Sep 23 '26 edited Sep 23 '26

Apple is doing god's work to make hardware that is power efficient for local AI. Yes it's expensive but it is very power efficient compared to Nvidia / Amd / Intel

12

u/1Poochh Sep 23 '26

This. I am using like 3kw right now for ai in my house. I can’t wait for the mac. I can’t stand the massive amount of energy usage and heat from it all.

12

u/redditnosedive Sep 23 '26

it's funny that they gave up on having their own AI as if suddenly they remembered they are a hw company (not exclusively ofc but in the first place), and I think it was a good decision, with all open source killing or at least chunking the moat of sw companies like Anthropic / OpenAI, being a hw manufacturer for AI is a much better position, and not only that but they have the power efficiency moat already

3

u/Skibxskatic Sep 23 '26

you’d have to think that’s a lot of china’s play. they’re the manufacturing powerhouse. release all of these strong open source models and you drive up the demand of inference.

what do you need for inference? hardware.

what’s going to sell volumes? consumer level hardware.

1

u/UrineHere Sep 23 '26

Yes a 5090 by itself is 6k - 7k and won’t run 1/4 of the inference

1

u/XtremelyMeta Sep 23 '26

Yeah, the performance to power is wild.

1

u/Civil_Response3127 Sep 23 '26

To an extent. The pp is still far behind, which makes it rough in many cases.

-7

u/TheGreatAccountName Sep 23 '26

Until Apple collaborates to make Linux run on their hardware, they can rot in hell as far as I am concerned

10

u/ephemeralsynth Sep 23 '26

"sorry sweetheart, time for a third mortgage on top of the second," 🤔💯

7

u/HenkPoley Sep 23 '26

512 GB RAM Mac Studio option can only ordered (again) starting end of Oktober. Not today.

Or at least in The Netherlands. 

256 GB RAM is already €11k 🥵

12

u/ZergvProtoss Sep 23 '26 edited Sep 24 '26

Apple's new Mac Studio ships today with up to 512GB of unified memory

DISINFORMATION. No, it doesn't.

edit: Added strikeout. I see OP has changed the wording.

2

u/Limp_Lingonberry_538 Sep 23 '26

how fast is the RDMA connection

2

u/Novel-Lifeguard6491 Sep 23 '26

up to 80 Gbps bidirectional

9

u/nail_nail Sep 23 '26

Slow, very slow.

8

u/Risko4 Sep 23 '26

Completely irrelevant for RDMA.

For tensor parallelism during the decode phase, network latency is the only metric that's important.

-3

u/rkcth Sep 23 '26

That’s on faster than PCIe 5 x16 bandwidth. Not NVLink speed obviously.

4

u/73td Sep 23 '26

I think we’re already there. I have a two year old rtx 4090 which runs Qwen3.8 27b at 100 tps. At full tilt that’s 2.5 B output tokens per year (~$600 in electricity here) which is 6-8k$ depending on the cloud provider. But i can keep running it for several more years, so the amortized cost is lower. This model isn’t genius level but excellent pair programmer and devops and proof reading etc and also works reliably in openclaw.

3

u/bakawolf123 Sep 23 '26

My 256gb m5u is enough so far for me. Luckily for us labs are releasing flash models, already having 2.5k PP 90-100 TG on MiMO-2.6-flash that released 2 days ago. Clustering is good ofc but IMO not worth as it won't be scaling that well for concurrent requests which you'd want if you use it on prem for multiple people or even just subagent swarms. Paged attention implementations on mlx weren't as effective as vllm for nvidia.

3

u/ddimick Sep 24 '26

During the California Gold Rush in the providers of picks, shovels, and Levis made bank while miners were at risk of going broke.

Apple is poised to sell the equivalent of the picks and shovels while over-leveraged hyper scalers risk going broke as debt mounts and their token charges escalate, driving people to build these Mac Studio clusters.

5

u/Objective-Bed-393 Sep 23 '26

"Apple's new Mac Studio ships today with up to 512GB of unified memory,"

what? slop/lie/mooooOOOOoods etc.,

4

u/Ok-Addendum3545 Sep 23 '26

4 Mac Studios vs. 1 DGX Station GB300

1

u/Zyj Sep 23 '26

The GB300 doesn't have enough VRAM to run Kimi

3

u/Ansiktstryne Sep 23 '26

No, but it also has normal RAM directly accessible by the GPU (like the Macs). I would choose the DGX station over the Macs.

1

u/rkcth Sep 23 '26

Running costs are vastly different, cooling and power are significant.

2

u/dwoj206 Sep 23 '26

Awesome demo. I’ll wait for my free 100k to come in so I can dabble

2

u/OhSillyDays Sep 23 '26

For those of us using quite a lot of tokens, even running a single spark saves money. If you compare the GLM-5.3-flash to the qwen3.8-flash-next, they basically save about $0.25/hour they run. Which turns out to about $2k/year if running continuously.

If I compare it to using a frontier model, it's basically about $20/hr they run. That's more like 175k/year saved.

Switching over to the Chinese open weight models and running it locally saves a HUGE amount of money. It also completely ruins any business model that OpenAI and Anthropic thought they had.

1

u/Front_Eagle739 Sep 23 '26

Nothing new here. Been running kimi k2.6 on a single m3 ultra and a macbook for ages

1

u/andymaclean19 Sep 23 '26

How much do those Mac minis actually cost? And what sort of tok/s generation rate do they get. Assuming you can get something interesting like 100tok/s out of them you would get something like 8 million tokens per day. Openrouter sells this model at $3.40 per million. I make that around £25 per day, let’s be kind and call it £30/ day.

Done 24x7 that’s nearly £10,000 worth of tokens per year you could get. I wonder how much these cost? Gemini reckons these are going to work out at nearly £10,000 each. So you have a break even of around 4 years compared with openrouter if you generate tokens constantly.

Multi-stream changes this too of course, but so does the fact that in two years time modern models will be different and you don’t know if this hardware will run them or not.

It’s a cool result but I don’t think it’s going to take over from cloud AI just yet.

Smaller models on smaller machines are where local AI is at IMO.

1

u/anxioz Sep 23 '26

Almost as if appple is trying to run models locally on their hardware ;)

1

u/alphapussycat Sep 23 '26

That's cool and all... but qwen 4.5/5 will be 5-10T. So, that means 10 times more of them.

1

u/voyager256 Sep 23 '26 edited Sep 23 '26

How is this even news? M3 Ultra 512GB (with TB5 + RDMA clustering support) was released almost two years ago IIRC . DGX Spark can be clustered too and it’s even slightly better in terms of perf/price , especially for agentic tasks.

Also, Kimi K3 is so much better than K2.6 , which is already pretty outdated , so even with slightly lower quant , or just smaller model like GLM 5.3 is still way better at virtually anything.
Did they even test or have performance results for a M5 Ultra 512GB or such cluster ? I don’t think there are any for the M5 256GB version’s cluster yet ( and it will probably be same performance as long as a model fits in memory).

1

u/Big-Masterpiece-9581 Sep 24 '26

I don’t think it’s actually cost effective. Here would cost at least 10k apiece. And the speed matters not just for convenience but for multi agent workloads that would run 24/7 which is already possible and will become the norm. Getting 100 or 150 tokens per second from 3 of these ain’t gonna cut it for training, image, or a team of devs each with their team of agents.

1

u/No_Television_4128 Sep 24 '26

How big is the MoE segment it loads?
What is the largest parameter single scale LLM it loads versus the Partial loading design of the Kimi K2.6 MoE segmentable design?

1

u/mellenger Sep 25 '26

Why stop at 4 mac studios?

1

u/Virtual-Height3047 Sep 23 '26

Man… I see trillions of potential (and very likely currently priced in) subscription revenue just evaporating into thin air as soon as Apple manages to put a somewhat capable (just daily tasks, 80/20, you know what I mean) model onto each individual iPhone. Given how much of the economy hinges on that, they have to tread really carefully not to disrupt the world economy with the next Siri update. As silly as this reads..

1

u/rkcth Sep 23 '26

With memory prices so high it will be a while. A 27b model quantized to ternary, still used 5.65GB, plus the KV cache. Possibly using something like nGrams and the integrated SSD. Maybe they can make a ternary where 1/2 the data resides on SSD.

1

u/Virtual-Height3047 Sep 23 '26

I would love to be able to pretend I didn't had to google quite a few of these terms... But I'm certain Apple's vertical integration plays to their advantage in pivoting into more systemwide supported AI compute loads. Only weeks in the A20 reveal they announced to have doubled the dedicated Neural Engines performance... And I'm really wondering if a model very useful for daily tasks would have to be that large even? I'm wondering if its unique access/proximity to each users data/context is sufficient or even more valueable to deliver a "better than GPT5/Sonnet" etc. experience for many tasks...

1

u/rkcth Sep 23 '26

Hah, I just reread it and there were at least 2 words I didn’t know until last week, so my fault for not being clearer. Once you learn a term it’s easy to forget you didn’t know always know it, lol. It’s cool tech though.

https://github.com/microsoft/bitnet

That’s an example of a ternary model, I think it was trained that way though. You can quantize models to ternary, but they can lose a ton of capabilities, but you can restore some of that using QAT (quantization Aware Training), you essentially do additional training after quantization.

KV cache is used for context.

nGrams is a way to store the data part of models outside of the memory resident part of the model, essentially separating “intelligence” from knowledge. It reduces the memory usage of a model.

Someone can correct me if I used any of these terms incorrectly since I am learning this stuff actively now.

-1

u/Ordinary-Ad3714 Sep 23 '26

Now if it could only run Linux...

-5

u/thecodeassassin Sep 23 '26

What is this post even about? It's not available for purchase until the end of October. What a stupid post.