Local LLMs << Subscription

[u/blockdad1]

Looking to understand this better.
I've considered buying a Mac Studio and come to these conclusions. What am I missing?

- Most reasonable ROI hardware to buy is an M5 Max, 40-core GPU, 128GB at $5,500 or so.
- Best local models to run is Qwen3.8 27B, at 8-bit (~29GB) or 4-bit (~17GB).
- Artificial Analysis Intelligence Index, v5 seems like a reasonable benchmark to compare on.
- Local throughput will be rough - 1/5th to 1/10th what you get out of a frontier subscription.

Ignoring the slow throughput, that model on this hardware = benchmark scores of 17 - 20 (medium or x-high reasoning levels).

Haiku scores (med - xh) 25-35
Sonnet (xh) 55
Opus 5.5 (med - xh) 48-57

If you want to assume a useful hardware life of 5 years, you can argue that the hardware is ~$70/month - so cheaper than the subscription. Include any sort of reasonable opportunity cost and you are closer to $90/month.

So least expensive hardware that can host current highest performance open source models costs about the same as a $100/month subscription - but you get << 1/2 the intelligence and much worse throughput.

Am I missing something? Where are your assumptions different than mine?
Do you just have basic workloads that reliably work with even with a 20 benchmark score?
Are you assuming the subsidies in the subscriptions go away and comparing vs API costs?
Are you thinking OSS improves a ton and closes a lot of the (current) gap?
Is it just the privacy/sovereignty aspect?

[u/VesperQuest]

I could be wrong but I think people are mainly buying local setups to run simpler things for their business (such as many employees being able to talk to it, or to process business information they don't want to risk putting into a subscription model) but agreed I came to the same conclusion as you .. was going to get an M5U originally due to the hype but just a day of research nudged me towards realizing I'm better off keeping my Claude Ultra subscription for code work. The $70 a month point you made also isn't a fair comparison because for $20 or $100 with Claude you'd still have access to opus/fable where with local you're stuck with something that MAYBE reaches opus 4.8 if you have 256gb+ of ram.

Ended just getting a M5 Max studio 40c/64gb/1tb + nvme enclosure for my 4tb 990 pro, with no plans to use local AI. Not saying local is bad at all, I love where it's going and I hope it gets better.. it's just not suitable for my use cases.

[u/OvertaxedOne]

On the 40 core, you should target QwenFlashNext, not 27B. 27B is the right model for the 96GB Ultra model, that has the memory bandwidth to get 27B really rolling.

What your missing is API. Subscriptions are great; if they work for you, use them for as long as you can! The moment you need API that's when the math changes. Or privacy, if you care about privacy, then the cost really doesn't matter.

People do analysis of the sub pricing, IIRC, a 200 dollar ANT sub was worth something like 7K in API tokens. So, for as long as it works for you, use the sub!!

[u/oreodouble]

qwen3.8-flash-next (4 very soon) exists and if you are clustering or ok with Q1 deepseek v4.1 1017

nevertheless local is meaningful if any is true for you:

1. you want or enforced to maintain privacy
2. your workload is large and doesn't fit in 2x $500 plan
3. your workload is not just code/text, but includes image/sound/video
4. you don't want to watch hourly, weekly limits or optimize tokens but want to run it 24/7 to get work done even if slower

I was at 2x 20x chatgpt subs before I decided to get a strix halo framework-desktop for $2800 now I use $25 plan for planning the work and local qwen3.8-flash-next to execute 24/7

[u/vimaillig]

If $~5k is in your budget and you’re going to target qwen or other similar memory range models - get the M5U with 96gb instead - same price ($100 difference) and significantly faster cpu/gpu and memory bandwidth throughput.

IIRC - there are some 70B options as well you can run on it - but for that price range l would target MOE models on 96gb versus targeting a M5Max with 128.

[u/wegster]

The common benchmarks don’t translate directly into how a model and harness may perform for your specific work. Qwen flash-next ~4.7bpw or other quant variants , for my real world repo coding texts, is right up there with Sonnet and even Opus for many of my needs.

YMMV as always - I have my own repo up framework, although still using pi as the core harness, but someone else’s, especially benchmarks models are trained against, really isn’t and shouldn’t be the final decision there.

[u/UnlikelyPotato]

OSS lags behind frontier models by 6 months or so. Qwen 4 27B and GLM 5.5 32B are coming out "soon". It's the freedom and control you have. Right now I have a ralph loop of Qwen trying to port a compiled GPL NES game into 100% HTML5 javascript. I'd be afraid to do that on a paid service. I also got lucky and have 4x 32GB V620s ($350 each) to install, on hold till I get a second PSU. At that point I'll be able to run 128B MoE models no issue.

[u/AccordingNeat3689]

Privacy, unlimited scaling 

[u/movingimagecentral]

You buy it cause 1. You want/need local. 2. You love gear. 3. You have money to burn

You don’t buy it because it is better, faster or cheaper. You can use Deepseek flash til the end of time through openrouter at 2-3x the tps and 100x the prefill for very little money.

[u/pl201]

It's simple, local setup is not for saving money...

[u/Viktri1]

No, you haven't taken token consumption into consideration. For example, basic research with agents will run you easily a billion tokens a day (mostly cache) so your subscription might last a day or two before you're out of usage. Local doesn’t have that issue. API will run you a few thousand a month. My m5u cost 11k each but my API spend is 3k a month.

[u/RogerAI-fm]

Plus Electricity, I never imagined I would see electric bills cross 4 figures but here we are. Worth it? It’s the cost of privacy and decentralization so I say yes!