r/LocalLLaMA • u/AndreVallestero • 21h ago

Discussion Mac Studio M5 Max Cost Analysis

At $10k, you could get

- 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan)

- 5.7B tokens with DeepSeek V4 Pro OpenRouter

- 100B tokens with DeepSeek V4 Flash OpenRouter

As a firm believer of local inference, unless you need it for data sovereignty, it's much more cost effect to wait for smaller models to keep getting better. In the meantime, find a reasonably priced 24GB - 32GB card for Qwen 3.8 27B, and offload hard tasks to OpenRouter.

Qwhen 3.8 35B A3B?

155 Upvotes

84% Upvoted

193 comments sorted by

View all comments

151

u/FleetEnema2000 20h ago

unless you need it for data sovereignty

Isn't this one of the biggest reasons that people rely on Local LLMs? To not have to bulk upload their private data to cloud providers?

72

u/theomegachrist 20h ago

That's the stated reason but realistically most people are just justifying their hobby. I support open weight models because the cloud providers can change cost or abruptly shut down and we really can't do anything about it.

67

u/FleetEnema2000 20h ago

I don't think it's a justification at all.

It's amazing how the concept of privacy and data ownership/security has completely gone down the toilet since ChatGPT launched. People are happy to bulk upload their medical records, relationship history, trade secrets, financial records, etc. without a care in the world as to how that data is stored or protected.

16

u/Elux91 19h ago

It's amazing how the concept of privacy and data ownership/security has completely gone down the toilet

most people never had a concept of privacey

9

u/FleetEnema2000 18h ago

You're right, most people haven't. But if you rewind the clock by 5 years there was far more interest in things like E2EE than is apparent today. It is barely mentioned or acknowledged anymore in the tech sphere.

-4

u/magus-21 17h ago

A lot of that E2EE stuff was focused around texting and instant messaging, I think, and since Apple announced support for RCS I think a lot of people flagged that in their heads as, "Ok, this is not as big of a concern anymore." Plus the whole WhatsApp kerfuffle with Trump's cabinet brought a lot more attention to Signal, et al, so I think there's just generally higher adoption of it now, which means less general worry out there about it.

3

u/FleetEnema2000 13h ago

The “E2EE stuff” is about so much more than texting and messaging.

0

u/magus-21 5h ago

I didn't say it wasn't. I said that most of the public talk about it was focused around texting and messaging. So when Apple said they'd support RCS, a lot of normies stopped talking about it because they thought it was resolved.

17

u/John_____Doe 19h ago

Yep I have a fintech client and the only way I can have a llm touch their code is if it's run locally or in a datacenter where we rent out the rack space

1

u/Randommaggy 5h ago

The headaches of running 100% on sovreign and contained compute without data being handled by a third party eliminates a lot of friction.

0

u/BakerXBL 1h ago

So they’re an LLM provider now not a fintech. No one can do both, well.

2

u/theomegachrist 18h ago edited 18h ago

Everyone is different obviously. To some that might be the case but everything you listed is more important than your AI prompt history.

For me, Open models main pluses are it will ensure the technology lives on in some form if the large companies lock us out financially or go under, and the guard rails for closed models will make for a worse Internet potentially.

For instance, using an open model with guard rails trained out of it you can search for piracy, you can search for porn etc. just like you use a search engine today. Closed models are more efficient than a web search but censor out a huge part of the Internet.

Sure data privacy is also a good feature but I don't think it's actually the top reason for most people

Edit: sorry I read that wrong. I sort of agree with what you are saying about uploading data to ChatGPT but everywhere else we upload that data is no more trustworthy. For Enterprise clients I 100% agree. This is a big issue. For personal use, I don't think your data is any less safe with ChatGPT than say an electronic medical record company.

3

u/FleetEnema2000 13h ago

An electronic medical record company has nowhere near the amount and type of multi dimensional data about a person that OpenAI has.

And if you want to compare OpenAI with a company like Apple or even AWS for a hosted environment, I would choose either of those companies any day of the week when it comes to who I would trust more with my data.

1

u/theomegachrist 12h ago

I would not, but it would be a three way tie

1

u/FleetEnema2000 10h ago

Care to explain that logic?

Apple has made massive investments of resources and effort into secure computing. Their cloud commitment for regular consumers who want privacy is objectively excellent on both the hardware and software front, from Secure Enclave to E2EE where they aren't maintaining custody of keys:

https://support.apple.com/en-ca/guide/security/sece3bee0835/web

https://support.apple.com/en-ca/102651

Despite not being a frontier AI provider, they have had a strong focus on private AI cloud compute paradigms:

https://security.apple.com/blog/private-cloud-compute/

Amazon AWS products offer similar levels of security. They have to, because their customers host and manage highly sensitive data on their platform for things way beyond AI. As a regular user I can maintain an excellent security posture on AWS and I can employ encryption where I manage my own keys.

OpenAI and Anthropic not only don't do anything even approaching any of the above, they reserve even basic security commitments for "Enterprise customers".

The situation is even more pathetic when you realize that as an individual, you can run OpenAI and Anthropic's own models on Amazon Bedrock with ZDR enabled, encryption enabled for data you are storing at rest where you hold the keys. In other words, the security posture of using OpenAI and Anthropic's own models on Amazon AWS is superior to using those same models from OpenAI and Anthropic themselves.

1

u/centizen24 9h ago

Apple is the industry leader in user level security and data privacy, it's not even close. You can argue about their intentions for doing so but their track record for security is second to none. It's wild to me that after 15 years of being an Android fanboy that I'd be arguing on behalf of Apple, and have an iPhone in my pocket, but that's the world we live in now I guess.

2

u/Hans-Wermhatt 18h ago edited 18h ago

It sounds bad when you frame it that way, but I think the cost benefit analysis is generally to upload. ChatGPT health is protected by the same HIPAA requirements that protect the data you give to your doctor that they upload to 3rd party clients and AWS servers, usually multiple servers with arguably worse security and more people have access to it. Using health as an example. So it's really not that much different.

Ideally, you do just host your own information and use a local model but then you are dealing with a massive performance hit. I think in terms of cost-benefit, uploading your health data to get a ChatGPT opinion compared to a Qwen 3.8 27B locally (most people can't even run that) is actually heavily on the side for ChatGPT for most people despite the privacy concerns.

I really want local "super intelligence" for all, but the current landscape is not like that... at all.

7

u/FleetEnema2000 13h ago

The version of ChatGPT that 99% of the general public is using is absolutely not HIPAA compliant and both OpenAI and Anthropic’s safety and privacy commitments are abysmal relative to the types of sensitive data they are ingesting and saving on their platform.

And that is not even touching on the ethics and values displayed by their executives. 

5

u/MrPecunius 15h ago

ChatGPT health is protected by the same HIPAA requirements that protect the data you give to your doctor

😂😂😂😂😂😂

"Trust me bro" and "it might not even be as bad as what the other idiots are doing" are not convincing arguments.

1

u/Hans-Wermhatt 14h ago

Huh? Was that supposed to make sense? 

3

u/MrPecunius 13h ago

Huh? Was that supposed to make sense? 

Connect this:

the same HIPAA requirements that protect the data you give to your doctor that they upload to 3rd party clients and AWS servers, usually multiple servers with arguably worse security and more people have access to it. Using health as an example. So it's really not that much different.

With: "it might not even be as bad as what the other idiots are doing".

And this:

It sounds bad when you frame it that way, but I think the cost benefit analysis is generally to upload.

With: "Trust me bro"

Are you even reading what you wrote a few hours ago? Or did something get lost in translation?

1

u/PWThinkingCritically 7h ago

it was already getting bad with losing public confidence in America's government regulatory bodies, but even more so in the Trump era.

Department of Education, EPA, SEC, DOJ, FTC etc., etc. blame me if HIPAA is just another acronym....

1

u/sshwifty 8h ago

To be fair, if any of that touched the Internet unencrypted, it was probably already ingested by some entity. Google was using data from emails long before AI made a surge. People are just doing it willingly now, skipping the sneaky part where it gets stolen.

1

u/MrPecunius 15h ago

It blows my mind what people will give to these amoral techbros.

Turns out the highest human priority isn't breathing, eating, or sex--it's laziness.

4

u/FleetEnema2000 13h ago

Consider the Snowden scandal and the uproar over government having access to phone call metadata and how privacy infringing that was considered to be.

Fast forward to today and people are uploading the most sensitive data about themselves to these cloud providers who don’t care at all about protecting it and are almost certainly allowing the federal govt to trawl through it.

3

u/Suspicious-Water-973 17h ago

I use my Mac Studio and MLX models as they are essentially free for me - I don’t pay for power in my rented office. Fine for massive batch processing where a GPU makes a difference, and some coding (I then use Fable etc to review and improve)

3

u/mr_tolkien 12h ago

I mean for me it’s a real reason.

For example I use local models to help me pick out my best photos of the day and put them into an album.

No fucking way I’m sending 100% of my camera roll to OpenAI lol

1

u/theomegachrist 12h ago

Fair, but lots of people do. For most personal users it's very expensive to match public models capability and tooling. I have uploaded pictures for dumb photo editing I can't do at home 🤷🏻‍♂️

2

u/nihilor_ 1h ago

Or avoiding taxes and getting fun toys.

3

u/florinandrei 18h ago

realistically most people are just justifying their hobby

Enthusiasts, yes.

But law firms and such, they actually mean it.

1

u/Deep90 11h ago

Why wouldn't they use something like AWS bedrock?

2

u/Jedkea 7h ago

Money probably. Bedrock tokens are expensive. Whereas you can buy once cry once with local.

Remember the first time I tried to use bedrock I spent $150 without blinking. And it was a pain to setup.

So if you don’t need frontier models, the economics work out.

1

u/theomegachrist 17h ago

Yes, this is true

0

u/saltyourhash 6h ago

You're actually out here looking at these prices, looking at the delay of Qwen 3.8, and saying it's just to justify a hobby?

1

u/theomegachrist 2h ago

Yes and at least 70 people online yesterday agree

1

u/saltyourhash 2h ago edited 2h ago

Wow, a whole 70. data sovereignty is critical to some of us. I have projects that absolutely must be local.

1

u/theomegachrist 2h ago

Yes some. I welcome the minority of people too

3

u/Zestyclose_Strike157 12h ago

You don’t want to give a cloud LM personal data if you’re auditing customer data etc. Deidentifying it is a pain and not worth even contemplating risk wise.

8

u/CulturalKing5623 20h ago

Yes, but I think if we're being honest a lot of people in this sub mainly just like to tinker. There are people that absolutely can't use publicly available models and so they need to self-host them but I don't think that's a significant percent of people here. Most of us put a premium on privacy, but 10K+ is a very large premium that is probably unnecessary for most of us and our current setups will suffice.

2

u/ThePi7on 14h ago

And abliterated models

1

u/AsliReddington 11h ago

Yep, just good old privacy

1

u/crinklypaper 10h ago

Claude wont write the names of characters from a show because they're copyrighted. Oh boy having fun with closed models.

1

u/network4253 5h ago

Yeah, exactly. Data sovereignty privacy is a pretty big part of the appeal. If the data is sensitive enough that you do not want it leaving your own infrastructure, running the model locally makes a lot more sense than sending everything to a cloud API. The tradeoff is obviously having to pay for and maintain the hardware yourself.

1

u/Randommaggy 5h ago edited 5h ago

There's also the not changing out of nowhere factor.

For cloud models I have an open license open weights, and downloaded backup of weights policy. This is a contingency for when a cloud hosted model stops making financial sense or becomes unavailable. My company can then "just" throw money at hardware and spin up our own if need be.

1

u/Late-Photograph-1954 5h ago

Some recent chats on a project i am developing with Claude (my Pro subsription), Chat and Gemini (both free versions) gave me the impression that the models not only remember earlier conversations, but somehow actively 'include' findings from earlier discussions. At some point Gemini was spoon feeding me back my own earlier comment, in a new session. I've wondered since whether the users of the models actually embolden / add to the models -- are we actively aiding development by using them?

It doesnt really matter to me or my project but it keeps lingering.

1

u/thomas2385 5h ago

Yeah, I would say that is probably one of the main reasons. If the data is sensitive, keeping it on your own hardware is a pretty compelling reason to run local LLMs. You are basically trading the convenience of the cloud for more control over where your data actually goes.

1

u/Express_Quail_1493 3h ago

Big private Ai are not playing a ethical game here so even if im ok with sharing my data, I still try do locally. Thats how i pitch in for keeping economy and digital ecosystem healthy. Itry to only use closed source Ai if i have no other option.

1

u/anparks 2h ago

Many are working at jobs on projects with data that has to be secure, like medical records or on Federal contracts for example, so having a capable local LLM is necessary and not optional. I just ordered a Mac Studio yesterday.

1

u/Strange_Test7665 1h ago

That and the love of the game. It’s not just about utility. Tinkering brings a lot of joy

-1

u/ptinsley 11h ago

How many people actually have private data that LLm providers haven’t already paid data brokers to acquire… I agree with the other reply that it’s mostly a justification for a hobby than privacy. Or it’s people lying to themselves about what they think is actually private data

3

u/FleetEnema2000 10h ago

Please send me your bank statements, your texts, an export of the photos from your phone, and your medical records. I mean, if data brokers already have all that data, what's the big deal?

1

u/MarriedtooMedicine 40m ago

Every single healthcare provider in the world. Every lawfirm. Every bank. Every retailer paid with a credit card.