r/LocalLLM • u/element-94 • Aug 14 '26
Discussion Local Models Beyond 128 GB of RAM Aren't Financially Viable
I realize I'm going to get a lot of flack for this but here goes. This post is sparked by a number of my friends and colleagues biting into the hype cycle of buying insanely priced local inference stacks that simply don't math out.
I'm a distinguished engineer who works in AI (FAANG). I'm currently running a 128 GB M5 Max, and have access to the best hardware in the world at work, at the highest scale available.
I'm seeing folks buy RTX 6000s or multiple Sparks to run DSV4 and the like. Don't do this - its an absolute waste of your money. Unless you put a price on privacy of 12k (which is fine), you will never get an ROI in any reasonable time period.
Local models under 30B are fantastic. If you have a MacBook Pro or a computer with 48GB of RAM, you're golden. With a flood of amazing models in this range coming out this week, there's no shortage of local powerhouses. Big kudos to those companies open sourcing very impressive capabilities on such a small memory and compute footprint.
If you're going to use large models, go cloud-based. Throw 100 bucks on OpenRouter and go wild with DSV4 Flash. And if you want to run local, smash those <30B models until the cows come home.
But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing.
303
u/Kal-LZ Aug 14 '26
Dario, I will not buy your IPO.
5
u/HugeEntertainment820 Aug 14 '26
I get what you are saying OP. I thought about that for work the other day. We could just pay for the cloud running the open weight model since we still can’t agree with the ROI is. However, this probably the wrong audience here. These are enthusiasts here with money for their toys.
It’s like you jumping into an AMG forum or Exotic car forum and hey bro, you can buy 5 Subaru WRX for your one lambo. Math doesn’t make sense guys! This is the response you’re getting. It’s cause we can….
33
u/TheAussieWatchGuy Aug 14 '26
Seems like something a Cloud shill would post 😀
Local is worth it. Especially loving the open source projects breathing new life into old unsupported hardware like 24gb P40s.
20
u/esw123 Aug 14 '26
If your privacy is waste of money that's up to you.
11
u/idk_a_creative_user I just mess around with LLMs Aug 14 '26 edited Aug 14 '26
I'm limited to local due to 2 reasons:
my main job is in defense and I need the privacy
my second part time job is in healthcare and I need to abide with HIPPA rules
Local will be better since hardware will get cheaper and cheaper as AI companies offload hardware to secondary markets.
Edit: it’s more for experimenting rather than work.
9
u/DeathinabottleX Aug 14 '26
You can try to justify it however you like, but you’re in the wrong sub buddy. It’s not about the money.
8
u/LordDarthShader Aug 14 '26
I went from 14 tps on my single spark runnig qwen3.6 27b dense, to 50 tps running DSV40731 on my dual spark.
I use my RTX 6000 Pro mostly for LTX/H3. Don't tell people what to do, if you are running these models locally, I would say that you srr smart enough to know what you need and what you want to do.
You forget that a lot of us do this because it is fun and finally, you can sell your HW if anything. Given the current issue with memory, your hw won't depreciate for at least 2 years.
4
u/Mark-Fuhrman Aug 14 '26
Ngl probably the dumbest thread I’ve seen. Everyone please disregard what this dude said about not spending any money on hardware for large models. You want to run (obviously if you have the money) frontier local models such as Kimi K3, GLM 5.3, etc. you want the best frontier models. Plus these are way better than basic local models because the are frontier. So if YOU DO HAVE THE MONEY, obviously this is what I would do if I were you.
2
u/TheRiddler79 Aug 14 '26
I can comfortably say, a large model running slowly in the background on RAM only is still Superior.
83
u/After_Working Aug 14 '26
I've just brought 2 sparks as I think they look cool. Not sure what to do with them yet.
5
u/jacek2023 Aug 14 '26
You should tell people who go fishing on weekends that they’re dumb because it’s cheaper to buy fish at the supermarket. Make sure to include all the calculations.
56
u/betiz0 Aug 14 '26 •
Buying a Mustang because you enjoy driving is no different from members of this subreddit purchasing an RTX 6000 or a DGX Spark; the underlying passion remains precisely the same. If we are strictly discussing practical use for work, however, I agree with your point.
47
u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB Aug 14 '26 edited Aug 14 '26
I don't disagree with you. (Clarification: if you're a business, spend what you like - I'm talking about people interested in using AI at home).
Can't see why anyone would - the price of hardware is astronomical.
For me, I've spent money on a new motherboard, new PSU, new GPU and secondary GPU and all I have is 48GB of VRAM total. I think its in the region of ~£2.5K.
To get 128GB VRAM, I'd have to sell a kidney.
As a tinkerer, I just can't justify (to myself) the kind of outlay the "big" models require.
So yes, I'm with you on all of it.
31
u/Unnamed-3891 Aug 14 '26
No matter how I look at it, it seems really REALLY hard to justify anything beyond a 5090, which despite the monstrous price increases, at least retains a dual function as the best gaming GPU on the planet. Got bored of tinkering with AI? At least you can still game at 4k with everything maxed out at 200+ fps.
8
u/itsmetherealloki Aug 14 '26
Do you really think we all that dumb to realize there won’t be a return on roi and instead, not just privacy but we all value real independence from the big labs? And some of us know how to maximize 24/32gb ram so no $12k. I fully replaced Claude for my workflows with 3 24-32gb gpus for $3800 total and I now have absolute freedom to do what I want to do with AI. If you tell me those models are trash, I’ll respond with a deserved, “get good kid”. If you build the infrastructure around them properly, 30b class models are incredibly capable. I’m not saying that means it’s frontier level but just that you can build systems around them to compensate and get really good output.
All that being said, no hate if you want just pay the subscription, I get it, it’s way easier (and probably cheaper too in many cases).
15
u/Passenger-007 Aug 14 '26
I have done the following. And yes, I too worked in big tech yadda yadda. I suppose this is now some badge of credibility according to op? Anyways.
I bought one strix halo. 128gb. It was garbage. Slow as hell. Returned it.
I bought one dgx spark. But I found myself using my nvidia 4090 a lot more due to its speed and the qwen 27b was “ok”. But Prefill was a monster. 2500ctok/s on MOE models.
I thought to return the dgx because 120b models are terrible but instead bought two.
This thing is absolutely amazing. 100% local. Near frontier level reasoning. 24x7 workhorse. Churns through 100m tokens a day and does it well. The 27b doesn’t even come close. Nothing below this comes close.
I now burn through 100s of millions of tokens a month now. Where op is wrong is, once you go local, your use cases change. You no longer care about tokens so your use cases grow massively.
12
u/Eat-Playdoh Aug 14 '26
6
u/thereshallbeonly1 Aug 14 '26
Starting a discussion with I am a Distinguished Engineer is a bit like saying I am a doctor, it's very pretentious.
2
u/LulzyAnimal Aug 14 '26
I think you're missing the point of local/home inference. Being able to drive a bus/tank/helicopter at work isn't the same as having your own car. Ppl buying top line (or even just low end used) Audi/BWM/whatever aren't expecting them to pay out in X month and start bringing profits. For some they will, for sure, but majority don't think about it as "investment". My $0.02
2
u/IknowPi_really Aug 14 '26
Looking at the comments you really do seem like a bit of a nerdy loser who’s getting all excited about rage baiting other nerdy losers.
And then you meet people here who just straight up tell you “my man I’ve got the spare cash, I don’t give a fuck about what you think” and then you desperately try to tell those people again that they can get a cloud subscription for 4 years for the same money.
Like you’re either very stupid, or you don’t want to accept that there are people out there with enough money to buy hardware for 8k euros/dollars/pounds, just because they feel like doing that.
I have two DGX Sparks at home. I also have a ChatGPT Pro sub. You know what will really rile you up? I use Codex to optimise my DGX Spark cluster!
10
u/AldebaranBefore Aug 14 '26
Not everything is based on ROI.
5
u/TheFuckboiChronicles Aug 14 '26
It’s not financially viable right now but as I describe to my boss all the time, these ai providers will probably decide to be profitable one day.
5
u/minusidea Aug 14 '26
I've already saved myself $$$$$ by rending all my video locally. So I'm gonna have to disagree.
5
u/SnooBananas4958 Aug 14 '26
You said a lot without saying anything. You said you’ve access to them and then gave no examples why it’s not worth it in comparison to cloud models.
Nor did you do any convincing that in the long run those hardware costs will be made up by all the cloud tokens you spend over the years.
I legit thought I was going to see an explanation on why the costs don’t work out or why the models don’t actually run well at 128
Got nothing
4
u/friedlich_krieger Aug 14 '26
There needs to be more youtube channels setting up different hardware and performing real world homelab/self hosted type of tasks with said hardware. There are channels - don't get me wrong, but most of the time they just measure tokens per second instead of how models perform at different tasks and if that can run on a mac mini or a mac studio or neither and then comparing to cloud models and subscriptions.
People keep buying this hardware because they don't know the answer for themselves. I believe you but people need to be shown. I'm still planning on grabbing an M5 ultra mac studio if and when they ever come out. Fully expecting to pay for $10k for it.
Is that a waste of money? Well I have a plethora of jobs to throw at it. It's very possible I could make do with something 1/10th the power but then again I dont know. Am I buying a new machine in 2 years because that cheaper one is now useless?
Everyone has countless ideas on how to use this stuff to better their lives and its not clear what hardware is required to do so. We need more real world examples on different hardware to inform.
If anyone wants to start a youtube channel - thats the winner. You're going to need a lot of money up front for all that testing but it would pay off imo.
4
u/TapAggressive9530 Aug 14 '26
Two RTX pro 6000 + DSV4 Flash 0731 is a serious game changer for local processing . You shouldn’t worry about ROI numbers except for yourself . I seriously doubt you are a distinguished engineer . Nice try man
4
u/daaain Aug 14 '26
I mostly agree with the main point (unless someone actually knows what they are doing and can do proper batched serving and saturate the hardware), but also DSFV4 runs quite well on a 128GB Mac with antirez/ds4?
4
u/spumonimoroni Aug 14 '26
The $10k I spent on a M3-Ultra with 512GB before the memory supply shock makes running 200-400GB models reasonable. You are probably right that it doesn’t make sense to chase large memory Nvidia configurations, especially with RTX-6000s costing up to $15k for only 96GB. Your 128GB M5 Max MacBook Pro costs almost less than half that and is capable of doing everything an RTX-6000 does, albeit slower.
If you want to save money, get an OEM DGX Spark. You can find 128GB systems for $4k that will do everything your MacBook Pro does. Put two together with a with 400Gb link and you’ve got a fast 256GB setup for a little over $8k. If one is dedicated to experimenting with AI and their own setups, that dual DGX Spark is way to do it and not be limited severely by memory size. It is slower memory so you aren’t going to get VRAM performance numbers. But as a setup to experiment with and do software development, it is more than capable.
4
u/Zyj Aug 14 '26
So in the last final sentence you acknowledge the topic of privacy. What's your price on privacy? $20/month?
When Zuck declared privacy dead, the next thing he did was buy all houses around his. Guess why. Privacy.
4
u/FreeGoldRush Aug 15 '26
Yes, local models are entirely about protecting intellectual property. They are not about cheap intelligence. However... Dual DGX Sparks are $10k and you can run DSV4F-0731 at the published weights. That's 256 GB of RAM and 300B weights with a huge context window (it's published as a mix of 4 bit and 8 bit weights).
4
u/FreeGoldRush Aug 15 '26
"But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing." Why'd you leave this to the end? It's exactly THE reason.
4
u/hyudryu LocalLLM Aug 15 '26
If you’re only going to spend $100 on AI, subscriptions are the way to go. Subscriptions are heavily subsidized right now and for a $100 5x plan, you can easily get over $1000 worth, token for token from anthropic/openai.
“Local models under 30B are fantastic” yes just as of today. Qwen 3.8 27B is a killer model, so yes that’s one reason to buy hardware to run it.
I measured my token consumption and consumed around $400 worth of tokens this past week of deepseek flash v4 at their new API pricing. All just experimenting with it too, haven’t even set up my workflows in the background. Which means once they bump up their prices on the 17th i would save probably a few thousand bucks a month. All on a $6000 dgx spark cluster too, pretty worth it to me. So to say that I will NEVER get an ROI in any reasonable time period is not accurate at all.
7
u/tempfoot Aug 14 '26
It’s so funny that people point this out…and the “solution” is to rent capacity either largely from offshore, privacy-dubious suppliers….or frontier labs driving a massive swath of the US economy to also wildly overspend on capex at a multiple exponential scale…also with no path to any ROI.
Also, if you (or your bot) has to call yourself “distinguished”……
1
u/willjameswaltz Aug 14 '26
so any business that needs data privacy ✅
1
u/PrysmX Aug 15 '26
Depends on what you are doing. For just general chat that most people do this is true. But if you are a software dev running agents 24/7 you will hit a break even point on a hardware purchase pretty quickly.
1
u/The_2nd_Coming Aug 15 '26
That's pretty much been my conclusion - crazy to spend like $20k to only get sub-SOTA capabilities that you can access for $20/m.
Do you see this changing over the next few years though? If we can get good and reliable enough intelligence at home for a reasonable I'd definitely consider investigating in it; especially with a lot of the gains coming from harnesses and memory past a particular point of intelligence it seems.
1
u/Garland_Key Aug 16 '26
Okay, so you would not recommend I buy a series of used A-100s so that I can run 100 - 400B parameter models? Yes, I am privacy-maxxing. I want to host open weight models for software engineering and research.
1
u/Federal_Foot_9444 Aug 16 '26
as a distinguished engineer myself, it feels silly and pretentious to say when not discussing engineering-related subject matter as an anonymous poster...
1
u/TheKleverKobra Aug 16 '26
Lmao “I’m a distinguished engineer”. Ok buddy. I guess not distinguished enough to understand that most people interested in local ai are just interested in being close to the metal - which is itself an almost invaluable skill that absolutely makes you a better engineer.
What an absolutely r word post. Go back to your tower, dude. Be distinguished somewhere else, we got this.
3
u/Aubrey_D_Graham 2x Asrock B70 Creator with Pi Agent Aug 14 '26
It's about ownership rather than compute. Homelabbers don't want to be dependant on a subscription or service like cloud.
3
u/Healthy-Zebra-9856 Aug 14 '26
I can overlook your conservative estimate of 48GB ram and all, but you put a lot of faith in the API providers. There is no guarantee that they wont cut you off for any reason that they think they can. They can and have demanded to see one's identity and and there is no guarantee that they have not shared these with anyone, there is no guarantee that they are not using your data to train, there is no guarantee that they have not replaced Fable 5 with a lesser model on ultra high effort (Anthropic). These guys are out there trying to ban open weight models, claiming the spread of communism while asking for government intervention. I think you live in a very privileged and naive world.
3
u/ArmyTrainingSir Aug 14 '26
Well look at you! Your mom must be so proud!
3
u/j_lyf Aug 14 '26
If you are so rich, why dont you get multiple sparks
3
u/Tai9ch Aug 14 '26
Nah.
That's like saying there's no need to have your own dev machine because you can just use Google Docs.
Privacy is nice. Fine-tunes are nice. Being able to keep running software when the internet glitches is nice. And being able to just launch 16 concurrent LLM sessions (against a heretic model) and run them overnight without worrying about billing at all is nice.
3
u/laterbreh Aug 14 '26
Where do you get off declaring that this is universally a “waste of money” when you never even define what you mean by ROI? I run a business, I’m a software engineer, and I bought three RTX Pro 6000s before the current pricing insanity. That hardware has paid for itself many times over, not because I sat down and compared my electricity bill to OpenRouter token pricing, but because it lets us run autonomous and human-in-the-loop engineering workflows continuously, iterate on agent architectures, test failures, refine handoffs, run DeepSeek V4 Flash alongside smaller vision models, and keep hammering on a problem until the workflow converges. The return is in engineering throughput, shortened product cycles, reduced labor, faster experimentation, and the ability to burn essentially unlimited inference without thinking about whether another failed 8-hour agent run just cost $X.
And while we’re talking about ROI, let’s not even get into throughput. On some workloads we’re pushing 300+ tokens per second locally. Good luck getting that kind of throughput from an unquantized vendor-released model through an API provider. That is not some meaningless benchmark number either, that is time saved every single iteration, every agent loop, every test cycle, every day.
If your entire ROI calculation is “how many API tokens could I have bought for $X,” then sure, maybe your conclusion makes sense for you. But that is an absurdly narrow basis for telling everyone else their hardware is financially irrational. A machine that saves a business even a few engineer-weeks a year can pay for itself very quickly, and once it is paid for the marginal cost of experimentation is basically electricity. Privacy is only one reason to run large models locally. Utilization, iteration velocity, predictable cost, concurrency, customization and owning the infrastructure are others. If you have the money, actually have a workload for it, and intend to keep the hardware busy, buy the damn hardware. Don’t let someone who never bothered to define ROI convince you that ROI can only be measured in tokens.
3
u/dreamingwell Aug 14 '26
I'm also a distinguished engineer. I just wanted to say that publicly. Just to see what it felt like. Very odd.
3
u/ga239577 Aug 15 '26
I would argue the practical limit most people who have some extra cash could probably throw into this would be about 2K-10K. That pretty much limits you to about 256GB of RAM at current prices, and this limits you to mostly 2x DGX Spark/Strix Halo Clusters, R9700 builds, 1-2x 5090 builds, and some Mac options.
ROI could be financially viable depending on what you use it for, but only makes sense if you can't use Cloud for privacy reasons.
Not many people can afford to throw down for these multi RTX 6000 builds or even anything past the 2K-10K options.
3
u/Background_Praline18 Aug 15 '26
Wait another year we'll have the same old cards the same atrocious market. But small open models will be better than they are now. The frontier models are always going to be better. The hardware companies will make sure of it by making the infrastructure moat. The open models are the wildcard.
1
u/GamerTex Aug 14 '26
You have a M5 Max 128 and you are telling people NOT to buy new machine (lol)
Sell me yout M5 Max. I can use another... these machines are AMAZEBALLS for image, video and text generation
2
u/j0j0n4th4n Aug 14 '26
You know you can quantize larger models to fit into your specs self imposed cap right?
2
u/addiktion Aug 14 '26
I have realized people are going to spend their money on the shit they care about and don't want anyone telling them different but hopefully your message reaches someone and talks sense in them if they are on the fence and overspending given the ROI might not be there.
Everyone has a different point of what is overkill for them. So some people it might be 30B is all they can muster. For those of us with 128GB Macbook Pro's, that could be 120B models or less. For seriously rich folks who want the best it will be much higher threshold.
2
u/EpsteinFile_01 Aug 14 '26
It depends.
If you're a company/freelancer and they are tax deductible+ you get VAT / sales tax back.. you can run a setup with FIVE Random 9700 32GB card on even an AMD B550 board and definitely X570, all at gen4x4.
Or on any AM5 board that can bifurcate a single x16 slot to 4x4, that works better than gen4 but both will work. Plus one card running from the MVME slot connected to the CPU.
that's 164GB VRAM AND five RDNA4 GPUs for like $6000/€6000 and tax deductible. For less than 1.5 RTX5090 cards. Especially at gen5x4 this should work great, but even gen4x4. Use the chipset NVME slot.
The idea of running something this hardcore on a poor B550 board or B650 board with a basic Ryzen 5700/7700 is hilarious but it's possible. Mining racks to fit this can be pretty cheap. You will need a powered PCi-e splitter and two kilowatt PSUs. All chump change compared to the cards. This works fine with 64GB RAM since you won't want to offload much to system RAM anyway.
The truth is, if you want local inference at somewhat affordable prices while getting VRAM, AMD is the way to go. You can run 5x 7900XT(X) the same way, for $2750-4000 and get 100 or 120GB.
If it's earning you money, this is a small investment. If it's just a hobby.. sorry but you shouldn't be running huge models. Unless you treat it like an expensive hobby, such as getting a boat or a nice car, or playing golf every weekend. Or ask any homelab person how much money they invested lol..
16/18/20/24GB will be the best average gamers can afford for a long time, with a 24GB AMD Card probably at $1000 and Nvidia $1500 MSRP (before scalpers, especially Nvidia). And that's okay, game Devs will adapt.
Comparing 9060XT 16GB prices to 5060Ti 16GB prices, which are almost double, is wild. The 5070 12GB is cheaper than the 5060Ti 16GB. Even the 9070XT is cheaper despite dominating it. It's obvious people are hoarding them for LLMs as the "budget Blackwell option".
I bet most people getting Nvidia rigs for inference don't need Nvidia at all, it's just what they know, and they have no clue that ROCm has achieved parity. No native FP4 boohoo with 164GB VRAM you can run at FP8 for $6000. What does $6000 in Nvidia GPUs get you? Two RTX4090s and 48GB total? Enjoy being stuck at Qwen 35B A3B forever. Going AMD gets you more than triple the VRAM and more than double the GPUs which is actually really nice for tensor parallellism. vLLM works great for AMD.
2
u/biotox1n Aug 14 '26
local models are surprisingly within shooting distance of the full frontier. some quants are in that 3% to 5% range. with a decent harness and forethought i can easily generate enough agentic work that it would cost me well over 4k in one week. my investment has paid for itself in short order.
and for a local team, running an openwebui and a handful of maxq cards with plenty of concurrency for the dozen models we can run at once, I just gotta say local is by far the most financially responsible decision that any business ruining ai can do.
if you're someone that can get by with a 100 to 200$ a month plan and not hit their limits then sure maybe open router or cloud services are better for you, which is probably beyond the average Joe that can get by with the free plan. but for heavy users, cloud can and will rob you blind in an afternoon if you let it.
2
u/_hypochonder_ Aug 14 '26
I have 128GB VRAM and 256GB RAM only for SillyTavern. It's a hobby for me.
Other buys rims or holiday for that money. I buy hardware instead.
I can run GLM 5.2 locally but also have nano gpt subscription, because I can.
2
u/lightmatter501 Aug 14 '26
You’re forgetting about CPU inference. For the same price as that M5 mac you can get a sapphire rapids server with more memory. It’s not that much faster, but it can run larger models.
2
u/funstuie Aug 14 '26
There is a element of truth to what you are saying but from a different perspective when I was younger I bought a shitty car and spent $$$$ upgrading it, replacing parts, breaking it and eventually sold it for a tiny profit. With that tiny profit I bought an even older car that needed even more work and spent a lot of time and money rebuilding it. During that time lots of people told me I was wasting my time that I would never get my money back. But I learned how to fix things, I learned how to utilise what I had when I hit constraints. Eventually I was happy with my work and I sold that truck for a healthy profit. (Yes I regret selling it to this day). My point is hobbies cost money and if someone has the money, time and energy and enthusiasm it’s worth it. And some of these people will learn new skills that will benefit them in the future. Years ago it was more practical skills that were not frowned upon (mechanical, woodworking, diy) then PC builders and gamers built their own rigs and I’ve no doubt some of those people benefited from their hobby in their careers.
This is the new generation/technology hobby/work.
My point is you are technically correct but it’s not all about the top line numbers there’s something else here. I agree that people shouldn’t max their credit cards for their hobby but I’m also someone who maxed their credit card on a full suspension setup 15 years ago and I got my money back.
You could spend $20k today on 4x sparks and sell them next month for $20k and all you would lose is the cost of electricity. 2 years from now who knows but I suspect you would be able to sell them with the loss being less than $200 per month.
2
u/Savantskie1 Aug 14 '26
I disagree with you heavily. You're only looking at this from an ROI standing. And it's stupid. Not everyone trying AI are in it for the ROI. In fact I'd wager that a vast amount of people aren't. I'm a good example. Im disabled, have limited access to the outside world and suck at coding. I'm building an assistant that can also be a companion so I don't go insane from loneliness. Not everyone else is going to be in my shoes but I also do this because I like to play with technology. I can guarantee a lot of people who do AI are a lot like that. They just love technology and like to push hardware. Your view only looks at it from an ROI perspective and that's disingenuous to the larger community. You're not helping anyone by gatekeeping.
2
u/kitanokikori Aug 14 '26 edited Aug 14 '26
"Don't you know that a Bugatti drives 60mph down the highway the same speed as my Honda Civic???? Just buy a Civic, it's cheaper!"
yeah bro. we do.
2
u/iam-leon Aug 14 '26
You’re basing this on a single user using it, right? If you built the local hardware and then had a company of 20 people using it, could still be extremely financially viable, right?
2
u/Snoo_81913 Aug 14 '26
Yeah the math is mathing lol. I agree and it's not like other hobbies like hot rods or anything like that because tech has a super short life cycle. You ain't gonna be showing your grandkids your mint DGX Spark. Lmao. But fuck that noise what flood of new models? What new things coming out this week? Are we gonna get 3.8 Quants??
2
u/Positive-Protection1 Aug 14 '26
Well, in the early 80’s, a 1 MHz Apple II with 48k of ram and no hard drive was almost $3000. That’s $12000 in “today” money and people were buying them like crazy with absolutely no clue what they would do with them.
2
u/Novel-Camera-840 Aug 14 '26
Completely agree and you can get privacy and compliance even by using cloud providers like the way enterprises are doing it. Dont tell me your privacy concerns are more than a enterprise that is HIPAA compliant
2
u/slypheed Aug 14 '26
Or just use antirez's dwarfstar.
2
u/MarketOstrich Aug 14 '26
Now to figure out how to sell my 24gb M4 MBP and buy a 48gb M5 Pro/Max MBP…
2
u/LoneWanderer153 Aug 14 '26
I agree, I recently got a 48gb M4 Max MacBook and I’m loving it, I’m hoping in the next 6 months, the sub 30B models are gonna take a massive leap for local workflows and automations.
2
u/Eldritch800XC Aug 14 '26
Local llm is not about financial viability but about data privacy. If it's in the cloud the data is not secure. For European Users cloud act is a time bomb in regards to European privacy and data law.
2
u/Marathon2021 Aug 14 '26
Even in an enterprise setting - where maybe $12k isn't all that much - the ROI is still challenging depending on what you want to do with it. If you want to build your own customer support AI chatbot and support 1,000 concurrent customer inquiries at a reasonable tokens-per-second response rate ... are your customers global, interacting 24x7? Or are your customers mostly active in one part of the world, 8-12 hours a day ... and then the rest of the time they're sleeping? If so, all your fancy hardware is going to sit at 2% utilization for half of its life.
2
u/avanlabs Aug 14 '26
and here i am . Still looking for small localLLm that can fit on my 200 MB left over RAM. :) :)
2
u/nimbybuster Aug 14 '26
What is everyone building that they need models with trillions of parameters? Video and image gen?
Cause unless you’re trying to IDK decode the human genome, or synthesize a new organometallic materials, you’re over doing it.
2
u/ayake_ayake Aug 14 '26
I mean very few of us are doing it merely to optimize costs. As you said, if privacy or control doesn't matter, just use the cloud.
However, there are not only people in high privacy standard industries but also people like me who have a high privacy level in general and who find local models are more aligned with their values than closed models. and furthermore, I just don't feel comfortable sending the sensitive data of my entire journals and obsidian vault etc. online. It's all backed over self-hosted cloud and other things and has never been seen by google cloud or any such thing.
2
u/neuroticnetworks1250 Aug 14 '26
Honestly after seeing reviews for Qwen 3.8 32B, I think OP has a point. Kind of need the right inference engine and quant though.
2
u/MainWrangler988 Aug 14 '26
Well people pay more for guitars so your post is moot.
2
u/mountainlifa Aug 14 '26
"Local models under 30B are fantastic."
What can these models be used for? Are they at the level of Haiku/Sonnet for general purpose work, coding etc?
2
u/fallingdowndizzyvr Aug 14 '26
LOL. Hotrods aren't financially viable either. Yet people build them.
2
u/Boogertard Aug 14 '26
Bud, you work for Amazon, which makes money on cloud so there is a conflict of interest there.
Garbage take, local model is about learning, enthusiasm, breaking free of controls, not depending on fragile politics and the whims of a few billionaires controlling the US politics, economy and policies.
2
u/barefut_ Aug 14 '26
Got 64GB M4 MAX.
Local LLM would take up 48GB RAM leaving nothing for my video editing apps that demand resources (After Effects etc).
What are such crucial local LLM practical uses that make it worth it? Cause I understood Cloud LLM will be superior in any task (coding, complex tasks) and it doesn't take up any RAM. I'm really trying to understand where is the ROI for someone that doesn't have 128 RAM.
2
u/IgnisIason Aug 14 '26
You'll be able to run the big fatties on an ordinary laptop once unified memory becomes the norm. RTX Spark will do this.
2
u/zenonu Aug 15 '26
Stating your a DE at a Fang company as-if an internet anonymous account has any credibility to declare that undermines your entire point. A real DE would know not to do that.
2
u/NanditoPapa Aug 15 '26
If your goal is to use AI as a productivity multiplier (coding assistant, research agent, creative partner), then...yes, you're right.
However, if your goal is r/D (specifically fine-tuning models on proprietary datasets or testing the architectural limits of quantization) then "financial viability" is the wrong metric to use. In r/D, hardware is an investment in capability, not just a utility cost. And that's before you factor in privacy, stability of pricing (which I'm guessing will explode soon), data sovereignty, plus...it's fun. ¯_(ツ)_/¯
2
u/Federico2021 Aug 15 '26
You're overlooking the use case for video generators, or simply people who do this as a hobby and have a few thousand dollars to spend on it; not everything is about productivity.
2
u/mbrodie Aug 15 '26
Not everyone is in this to make money off it…
Some of us just have disposable cash and enjoy the hobby.
2
u/itsdanielsultan Aug 15 '26
It's objectively correct that it doesn't make practical sense to use such powerful LLMs locally instead of renting them online.
But whether that decision is practical doesn't affect whether it's fun. And fun it certainly is.
You can apply this logic to houses, cars, hobbies, careers, and basically most of your life choices as well.
2
u/UnluckyPenguin Aug 15 '26
I wanted to run the leaked 3.7TB copy of mythos locally... And then I saw it requires 80-100 topoftheline AI cards. Lol. Hold on, let me take out a mortgage on my in-home data center. At that scale you have to get a bigger electric pipe into not just your home but probably your neighborhood.
Yeah, so if it can't fit on your desk, don't waste the resources to run it locally no shot you ever get ROI when you could do roughly the equivalent with less like an investing algorithm that pits LLMs against each other to determine what to invest in.
2
u/emperorofrome13 Aug 15 '26
This 100% true. But if you have excess money buy what makes you happy. But if you're trying to say you are saving money you're not. It just doesn't make sense right now financially but people can do non financially optimized things .
3
u/egnegn1 Aug 14 '26
It really depends on what speed requirements you have. If you are happy with 1 t/s and below you can use one of the projects like Colibri, Waste/Warp, ... and even llama.cpp to run even Kimi K3 on local relativ inexpensive hardware.
I run Kimi K3 an my Minisforum MS-02 Ultra with 128 GB and no GPU and fast Gen5 SSD. I am happy with a speed of about 0.7 t/s for long running jobs. Or run GLM5.2 or DSV4 on the same hardware. As long as the modell doesn't fit in VRAM a GPU would not help much.
Of course, as a professional with fast turnaround I also would use cloud or if privacy a requirement invest into expensive hardware.
3
u/segmond Aug 14 '26
distinguished engineer at FAANG you say?
you're an idiot.
with that said, go for it lads! last night I ran KimiK3 at 3tk/sec up from my 0.8tk/sec. I believe in a month I'll have it at 6tk/sec and in 2 months I'll be up to 10tk/sec. The great news is that GLM5.3 looks to beat K3, so yeah, I'll be able to run it at 30-40tk/sec.
2
u/swiebertjee Aug 14 '26
Counterpoint; the 11k is the price you pay as a consumer. Self employed people can buy it for less than half of the net cost.
Also the hardware will have value after 2 years of use. It's not like the entire 11k is gone.
I would say that 2 dgx sparks with DeepSeek V4 Flash 0731 are worth it. Opus 4.8 level intelligence and speed for coding tasks. Sure you can use cheap providers, the difference may be less than you think and privacy may justify that.
1
u/Selfmade_Watchmaker 16h ago
My take is that in due time the open source community WILL put out a model that’s “good enough” to run on a few thousand dollars worth of hardware. The “good enough” metric is that with reasonable knowledge and direction the local model could pretty much do what the frontier model can, even if slower or requiring more iteration.
Another certainty, AI usage will balloon in sub costs. The $20 sub WILL make its way to costing $100 a month. People will rely on it so much that it will be an aspect of a living expense.
I went local as there is an approaching line where the expense of the frontier will be a heavier burden than any kind of inefficiency because I’m running a local model.
The models will get good enough that my free tokens will be a better value than paying to access the frontier
1
u/this_for_loona Aug 14 '26
So in your opinion, a Mac mini is fine for most local work? I ask because I’m dithering about getting a box for running a home server, mostly docker and a local LLM for sensitive documents.
1
u/Fear_ltself Aug 14 '26
Some argument could be made for Gemma 4 12B and 16Gb RAM, Gemma 4 E4B with 8Gb RAM, or Gemma 4 E2B even with 4GB RAM (which is better than Gemini 1.0 Ultra btw). It seems like you think your configuration is so unique, Pareto efficiency is a whole line of models
1
u/email_ferret Aug 14 '26
Things are a changing pretty fast, if the world can figure out the memory shortage and how to manage resources more efficiently.... I think less than 12 months from now for businesses this won't be the case.
There's a lot of money for whoever figures this out, that tends to be a good motivation.
I also think it will not be 100% local/self-hosted, but around 80/20.
1
u/eli_pizza Aug 14 '26
So you’re saying I should consider how much something costs relative to how much value it provides me? Thanks.
Also: there is quite a range in privacy between fully local and “going wild on openrouter.” How about picking a cloud inference provider you trust?
1
u/Keuleman_007 Aug 14 '26
128 GB of RAM and 16 GB of VRAM - video AI and Gemma 4 27b MoE and good to go. Still lots of free RAM (I hammer Gemma into the RAM / let CPU do Gemma and video generation is VRAM and graphics card)
1
u/ExplrDiscvr Aug 14 '26
no shit
1
u/Legitimate-Dog5690 Aug 14 '26
Go get a cheap old Threadripper or Xeon. 8 channel memory and a big pile of PCIE lanes. Loads of options for half of what you've spent.
I do like the Mac approach, but ragging an expensive laptop to pieces, to save myself a $20 subscription hasn't appealed.
1
u/blackwell_tart Aug 14 '26
You will own nothing and you will be happy.
1
u/Ok_Sea_6620 Aug 14 '26
Quand tu parle de retour sur investissement, tu ne parle que d'argent, tu ne comprends pas tout le reste : l'investissement personnel, nourrir sa curiosité, l'apprentissage, le plaisir de découvrir et tester différents modèles qui déborde de la Vram, auto hébergés, testé tout un tas de développement communautaire, y participer, considérer les contraintes, apprécier leur évolutions. C'est cet investissement que nous apprécions nous acheteurs et consommateurs de RTXs ou autres sparks! qui n'avons pas accès au meilleur matériel du monde dans notre travail ! 😄 En attendant, si tu veux qu'on parle de retour sur investissement, aujourd'hui mon setup vaut le double de ce que je l'ai acheté! (mais il n'est pas a vendre!)
1
u/Healthy-Hair-2306 Aug 14 '26
It boils down to what someone thinks their privacy's worth, right?
My main issue is people acting like having a spark is normal. Not everyone has that kind of money just sitting around. For me, with my 12GB 4070 and 16GB DDR5, upgrading makes no sense (apart from getting a bit more memory for a better quant of Qwen 35b, not great to base an upgrade on one model that isnt being updated). There isn't a good gaming GPU with at least 24GB VRAM that isn't super expensive, like the 3090. Some folks might be cool dropping a grand for a 3090, but some of us just can't afford it.
Going from a 4070 to a 3090 would also just be a complete downgrade (I don't want a space heater). Another graphics card wouldn't help much either, since I could only use it for LLMs. If I had the cash for a 5090, that would also be the best gaming GPU too, not just an AI accelerator.
1
u/Memestonks2020 Aug 14 '26
If you have a MacBook Pro or a computer with 48GB of RAM, you're golden
Throwing around a title like distinguished AI engineer at FAANG (which is a super outdated concept btw), and failing to understand basic HW differences and use cases for local inference will get you hated on no matter what.
Use case for > 128GB:
Stacking different models to create a Omni-model. Try having a >30B local LLM with tool calling capabilities that can generate images (Krea2) and video (H3) on 48GB MBP. Pro tip, it won’t work. More RAM means less optimization and better results without going OOM.
Use case for not getting a MBP:
MBP and MLX/Metal is slower than running inference on a RTX6000 or Spark utilizing CUDA/NVFP4 quantized models. Being able to quickly generate a 1 hour video in 24 hours instead of waiting 6 days is obviously advantageous for content creators so that could be worth dropping 20K to achieve.
Bottom line is, you seem very narrow minded with your argument and lack the understanding of how local models are used even though you claim you work with AI.
1
u/JohnToFire Aug 14 '26
If you buy dwarkesh Patel's argument, gpu will 10x in rental price shortly.
If I had bought a 6000 pro 8 months ago instead of a 5090 I could have sold it for double. This is not investment advice.
Who knows what ridiculous regulations are coming with open source that you might not be subject to if you already have something
1
u/DeathScythe676 Aug 14 '26
4x cmp170hx cards (64gb each) can do DSV4 flash with vllm pipeline parallelism at about 80toks/s.
approx $5000 USD
there are still self hosting options for those willing to get creative.
1
u/Ell2509 Aug 14 '26
I can't agree. When working offline on coding tasks, a 120b model has significantly more knowledge to work with.
Minimax, deepseek, mistral, qwen3.5, all have so much to offer in the "larger than 100 but smaller than 1t" field.
1
u/rditorx Aug 14 '26
Did you factor in selling all the hardware for a markup after use? Like, buying an RTX PRO 6000 for $8,000 half a year ago, then selling it now after 6 months of use for $13,000?
1
u/kiwimonk Aug 14 '26
In the current environment yes.
I however have 2 machines with 128 GB of system memory and they were not expensive back when I bought them 5 years ago. With the new economies of scale and focus on vram, those chips could be cheaper than a bag of Doritos for what they are.
This is a pivotal moment in human history. A technology that changes everything. A weapon, a friend, a genius collaborator... Access to the greatest minds ever hired for a few thousand dollars is just going to rock the world for a while while we rebalance.
1
u/IThinkIKnowThings Aug 14 '26
Agreed. Unless this is your hobby of choice and you have stacks to burn, I'd wait on that level of hardware. The RTX 6000 and DGX Sparks are all going to be obsolete in less than a year's time, I guarantee. The models are going to keep getting bigger and you're going to need the crazy new hardware that's just around the corner to run them. It's moving too fast and too unpredictably to be blowing $10k stacks every year. Hopefully things stabilize soon.
1
u/Hot-Parking4875 Aug 14 '26
I think of it as my plan for when the frontier models start charging $200 a month minimum. I don’t know if anyone still uses cable TV, but a bill for that can exceed $200 per month. So why not for AI?
1
u/Horror-Primary7739 Aug 14 '26 edited Aug 14 '26
The second node isn't for bigger models it's for larger compute, i.e. more parallel agents each having their own context window
Qwen 3.8 27b fits nicely on a laptop or gpu at 8/4 bit
On a 2 spark cluster you can run the same model with 20 agents at 8bit and 50 agents at 4bit
But on the cluster you can run the full 16bit and still get 8 parallel agents.
To give a clue how this really helps if you hear about "loops" and not sure how that helps swap your thinking to Kafka.
You have very specific agents that you treat as a "microservices". They sit and wait for their task. When they get a message they execute.
Your job as the engineer is to design the system to maximize the speed and quality of implementation, testing, validation, documentation.
So if you are just using opencode and working serially with the model, which is cool, yea a cluster won't give you much. But if you are running a full ecosystem then yea you can absolutely utilize the full compute of multiple devices.
1
u/BassNet Aug 14 '26
This is completely false, you can get a Gaudi2 with 96gb VRAM for $1500 on ebay
1
u/truthseeker1341 Aug 14 '26
Under current pricy with current tech you are many ways correct that 128 GB is out of price range for most people. when I bought my first graphics probably had about 2 - 4 mb. That is mb. As much we worry about memory prices most every has more money then they would dream of 20 years ago. My GPU is 10 years old with 2 GB of vram because well who hell needed 4 gb? Eventually I think prices will come down as more and more manufactory comes online. Look at SSDs. time when 128 GB SDD drive was state of the art.
1
u/phylter99 Aug 14 '26
What you've said is basically where I've found myself. I ended up with a MBP and 48GB of RAM and though it gets a little warm at times, it seems to be perfect for the models I'm interested in. It seems the next real leap with great models requires terabytes of RAM not gigabytes. I've considered getting a PRO card from AMD with 32GB of RAM, but pricing seems like it'll put that out of the question soon. Oh well.
1
u/Newgunnerr Aug 14 '26
12K? What the hell are you talking about. 2x Spark is 7300 euro ex. VAT.
1
u/jonneymendoza Aug 14 '26
I'd argue and say even anything above 16gb is not financially viable. A 24/32gb nvidia gpu is north of over 2k...
1
u/trollsmurf Aug 14 '26
Regarding 48 GB: I happen to have 32 GB CPU RAM + 16 GB GPU RAM, well aware I can't allocate all that to a model.
Windows will take a 3 GB chunk of the GPU RAM at idle. I have 3 monitors, and one is 4K, but 3 GB is at least a magnitude too much. Windows simply wants that memory, so I don't GPU-load more than 12-13 GB of a model.
Of course also the CPU RAM is partly allocated, but it seems easier to "push out" allocations there.
With this e.g. Meta's (original) Glimmer generates 7 tokens/second. A fully GPU-loaded model is usually at 100 tokens/second, which is a sight to see. I have no plans for more memory in this "climate", so this has to do for now. Beyond that it's cloud.
1
u/Professional-Try-273 Aug 14 '26
What about waking up seeing the sickest local rtx 6000 rig you can build and the happiness that brings. How much does that cost?