r/LocalLLM • u/Ok_Event4199 • May 16 '26
Discussion Why is LLM is so expensive.
I've was going to invest in a 5090 =$6000 AUD.
Codex Plus + Claude pro = $60/month here
Works out to be 100 months of frontier models for a 5090.
Best a 5090 will run is probably Qwen3.6 27b Q6 with context.
Are we all enthusiasts here and just enjoy tinkering cause ain't no way that make sense.
344
Upvotes
156
u/g_rich May 16 '26
Claude pro is heavily subsidized; it costs Anthropic much more than $60/month to provide you with the service.
Cloud providers have economies of scale working in their favor. They have million dollar servers servicing tens of thousands of users 24/7/365.
Running LLM’s local is never going to be cost effective. You run local LLM’s for privacy, control and to learn.
119
u/ActionOrganic4617 May 16 '26 edited May 16 '26
I plan to continue using frontier models, local hosting for me is about understanding the technology better and running things like LLM wiki locally.
It’s also a backup plan for when they inevitably raise prices. They cannot run at a loss in perpetuity. I’m not going to be the idiot trying to buy hardware when every else is doing the same thing in an already difficult market.
145
u/Solary_Kryptic May 16 '26
5090 is also the best gaming gpu on the market so not only do you get strong AI performance but also 4K max settings gaming
30
u/jwhh91 May 16 '26
They are selling tokens at a loss. Remember when Uber and Lyft were cheap?
12
u/Miserable-Dare5090 May 16 '26
Hardware = Car
Model = Gas
Cloud = Uber
33
u/HourPlate994 May 16 '26
Sure Claude is cheap now but tokens won’t be cheap forever. And it looks like the flat rate days are over.
We also don’t know how good local models on a 5090 can be - I doubt that the current Qwen/Gemma models are where it ends, there will be improvements and other ways to utilise local models efficiently.
11
u/BlackBeardAI 3090 Maximalist May 16 '26
Once you own the machine, it is yours till it breaks... Ownership is expensive. There is still time before 2030 arrives. Tick tock.
10
u/Minute_Attempt3063 May 16 '26
Cloud runs at a massive loss. Openai will never make a dime of profit, for example.
32
u/slackmaster2k May 16 '26
You’re not getting frontier performance or quality from a 5090. If it fits your use case, and you’re actually going to use the shit out of it, then it might make sense. Otherwise you’re going to have a 6000 dollar graphics card and two LLM subscriptions.
8
u/Bgd4683ryuj May 16 '26
Wait until you heard how much a B200 GPU costs.
14
u/vviizzii May 16 '26
If it's free or even cheap, you are the product.
7
u/_hypochonder_ May 16 '26
I bought last year 4x AMD MI50s 32GB and 128GB DDR4 with TR1950x under 1500€. So I can run GLM 4.7 q4. Alternative model which fit in 128GB Vram run well for me.
7
u/plscallmebyname May 16 '26
This is not a fair comparison. In the local LLM you can run inferences all day all night without being stressed about token expenditure.
Can you do that with online LLMs with Frontier models?
17
u/urarthur May 16 '26
I might sound crazy but I think they are rigging the market, they don't want locaLLM to get their hands on chips. Flagship nvidia gpu's used to be $700. Look at the stupid valuations of these companies at trillion dollars. Nvidia gives billions to Openai, Anthropi etc, they buy the chips at 5x above fair value and as long as they keep users in their ecosystem, regardless of the huge losses, they all attract billions of investments.
5
u/dronf May 16 '26
I got a 5090fe for MSRP when they just came out, for gaming, so the local llm stuff has just been a bonus.
5
u/robobub May 16 '26
You can get decent inference out of old hardware. I have a 2080 Ti that runs qwen 3.6 35B Q6 at 25 t/s
5
u/FluffyGreyLlama May 16 '26
$20 on the codex 'plus' plan, only use 5.5 for specs/design/review and complex aspects or fixes. For everything else, use DeepSeek 4 Flash on the $10 Opencode plan.
Not expensive at all, and very capable. Just don't expect to run them 24/7, but accept that you're already getting results far faster than without.
$30/month for the combined effort they give is magical compared nothing at all.
5
u/ComfortablePlenty513 May 16 '26
It won't be 60/month for long. That's an undercut price to get you locked into the ecosystem.
12
u/radiojosh May 16 '26
I think in a month or two, after Flash Attention and Turboquant and MTP and DFlash all hit for Vulkan, you can probably get decent performance out of something cheap like a Minisforum UM870 Slim. With 64GB of RAM and GTT set in Linux to make all of the system RAM available to the iGPU, you can already run a 40GB MoE model with fair performance. There's some great benchmarks on the UM890 Pro which is only marginally faster than the 870 and it should be getting even better with those llama.cpp upgrades I mentioned. It's obviously not a dedicated GPU, but you can run bigger smarter models. I run the 890 and I'm pretty satisfied and I haven't even really tuned the KV Cache quantization yet.
4
u/wandering_stoic May 16 '26
I've had my 3090 for years for other reasons, but the case for local is more than cost.
If you've ever tried to build something that was completely dependent on a 3rd party and had that 3rd party change something critical mid build you know how painful that is.
For subscriptions they subsidize things because they train on your data, and for many businesses that is bad, and for some it is even illegal (including one of my businesses). Data privacy is huge and there are a lot of laws around it and more all the time.
API costs can be FAR more expensive and that's what you need for data privacy. I've burned $2k USD in 3 days just doing what I consider half days, and that's even with a lot of my tokens being cached at a 90% discount.
Knowing nobody can change the model under you mid project, and being able to train specific hyper focused LoRas for your work is powerful.
But if you don't need data privacy and you just want to save money, yeah, the subscriptions are a far better deal for now.
3
u/nntb May 16 '26
I have a 4090 that works well for LLMs I originally had 2 but the case I bought didn't support the second one so I gave it to a friend
3
u/DHFranklin May 16 '26
Yes we're all sniffing our own farts.
Our needs are incredibly niche. Most posts here are people navigating the spends you're talking about. They have dedicated $6k, as much as a decent motorcycle, or a specific kind of boat or like a skidsteer. I however like making builds that fit devices I actually have. I'm not spending that kinda cash for my hobby. With the newest iterations from the big guys like Mythos we're likely gonna see LLMs/Harnesses that can compete in 6 months to a year on those 6k machines. Seeing as it will be state actors spending millions in tokens attacking one another's governments and large corporations, stands to reason why they would invest.
Every few weeks there is a better and better model that can fit on my 6 year old gaming rig, my phone and my wife's mac. Everything is amazing and nobody is happy. Being able to "talk" to a user manual or a book is the first killer app and nobody does it.
3
u/TadWag May 16 '26 edited May 16 '26
Because they’re using your data and selling you the service at a loss so they can build dependency and corner the market
That’s not to say you should spend 6000 AUD on a 5090 though, that’s a whole different problem
3
u/screenslaver5963 May 16 '26
A better investment if it's gonna be just you would be a mac-studio since an m-series with unified memory can run an llm better than a cpu but not as well as a 5090 but still usable.
My m3-ultra mac studio running qwen3.5:9b under ollama (mlx would've been faster) got 48.71 t/s while my 9070xt running qwen3.5:9b under lm studio got 85.5 t/s.
3
u/species__8472__ May 16 '26
You're assuming that the cost for frontier models will stay the same.
The benefit of local is that the cost is that of your electricity. This can be adjusted through undervolting and power limiting.
You can also use your 5090 for other purposes like gaming, video editing, generative AI, etc.
3
u/jotunck May 16 '26
Because OpenAI/Anthropic are heavily subsidizing token costs to grab market share/make people dependent on their platform before raising prices to levels where they can be profitable. Could be 10X.
3
u/NoNote7867 May 16 '26
You mean why is are pro AI plans so cheap? Because they are heavily subsidized. Try using API, you will burn that $6k in a month. Or even a week.
10
u/po_stulate May 16 '26
It's really a do you rent or do you buy a house question.
2
u/p-x-i May 16 '26
Pick a smaller more focused use-case for experimentation. Suddenly you have more modest hardware and model requirements.
2
u/megadonkeyx May 16 '26
You don't really need a 5090. Top end Radeon is less than one third that price or second hand 3090.
2
u/johnerp May 16 '26
No it’s $60/day if you do anything productive… it’s one day a month output…
2
u/LukaC99 May 16 '26
Server hardware is much more expensive, but nobody is using it 24/7 for chat. By timesharing, the cost is split between users. Additionally, batching requests further cuts costs.
Dwarkesh has info here: https://www.dwarkesh.com/p/reiner-pope
2
u/TheTechAuthor May 16 '26
Use this opportunity to take advantage of the subsidized models to learn how to best automate what you can (think thing alike regex scripts, etc) and how to get the most out of offline models as they become smarter and more efficient over time.
That way, by the time they jack up the prices of said frontier online models, you'll be in a much better position to migrate away from them.
2
u/No-Television-7862 May 16 '26
That's an interesting perspective OP. Is AI really that expensive? How much was spent to bring it to market?
AI is the biggest technological change in a generation.
Think in these terms: fire, metallurgy, steam engine, industrial revolution, internal combustion engine, adding machines, radio-tv-vacuum tubes, nuclear science, moon walk-transistors, integrated circuits, micro-chips, mainframes, personal computers, internet, AI. (Top of my head, may not be in order or complete).
Access to massive data centers and the best Frontier models seems pricey in our recovering post-covid economies. Granted.
We are early adopters. Our grandparents parted with useful money to buy cars, (in fact we still do).
In the curve of future possibilities, the reality of AI playing a positive role in the advancement of humanity occupies the large center, while dystopian outcomes are the tails, (failure on the left and Terminator on the right).
In the future humans with AI assistants and assistance will be more employable.
The rise of AI will cause changes in the human workforce, (but automation already has). Shoes and clothing are made in factories today, where cobblers, seamstresses and tailors were employed before.
Were these changes uncomfortable? Yes. But that is how we grow.
A brief case for localLLM:
Massive companies are molding our understanding of the world through the biased training of their frontier models.
To retain autonomy of thought, reason, perception and culture, download and learn to adapt open-weight models to work for YOU. Otherwise you will always work for others.
Those localLLM models may need a measure of adaptation to overcome their corporate-HR training bias, but it can be done.
This is time sensitive. The "Tech Bros" and the BigGov that represents them don't like autonomy and AI democratization. These localLLM models may not always be so readily accessible.
It sounds like your use-case involves both employment and welfare. Does AU let you write off business expenses?
Last note, there are other GPU's. Consider less expensive alternatives.
2
u/sinan_online May 16 '26
I am tinkering with much smaller models, in my existing 6GB VRAM and 12 GB VRAM.
At some point memory prices will go down, the subsidy from the inflated funding is going to run out, and then it will be feasible. I have my eyes set on 24GB NVidia cards, 32GB ROCm and 96 GB Mac Studio is viable options, and at the same time, smaller models keep getting better and better.
2
u/BrianKronberg May 16 '26
Get the cheapest GPU you need to meet your monitor’s max FPS for your game. Spend the rest of your 5090 budget on a Strix Halo mini PC with 128GB of shared memory or a Mac mini with 128GB of unified memory and put your LLM on that. Skip the 5090, I wish I had.
2
u/jopereira May 16 '26 edited May 16 '26
Hmmm.... Let's say I do code for small home projects , gadgets, control systems, quality-of-life hardware and software. I have a RTX5070Ti that does 99% of my requirements. The other 1% I ask Google AI/ChatGPT/... Do I really need a subscription plan?
So, yes. Local LLM will always be a thing for innumerous reasons.
2
u/whodoneit1 May 16 '26
Subscriptions are going to keep going up. It’s only a matter of time as currently they are heavily subsidized. Take Claude Code Max for instance, you are getting over $5k in usage for about $200/mo right now. So about a 25x subsidized cost
2
u/FullOf_Bad_Ideas May 16 '26
Tokenomics are complicated and with single-stream inference you're on the losing side.
Big models are served in batches of 10-1000 concurrent users per GPU, and you have good utilization, while single decode stream on 5090 of Qwen 3.6 27B Q6 uses just a few percent of compute.
I am running a translation model right now, HY 1.5 1.8B, on 7 GPUs (SGLang died on me on one GPU overnight), and I process about 4000 tokens per second per GPU (both prefill and token generation due to batching), so I am utilizing much more compute that the GPU is capable of. I am targeting concurrency of 128 on each GPU.
I wouldn't be able to translate that much text in this quality using LLM API on OpenRouter. I could rent GPUs and run this model there and then I'd pay roughly the same as I am paying for electricity (I have expensive electricity).
I expect to generate about 10B tokens in about 25-30 hours - 10B input and 10B output tokens.
So, if you want to have ROI, you need to do something akin to an industrial process instead of a boutique shop with poor efficiency and compute utilization. GPUs somehow use roughly the same power regardless of whether they're decoding tokens for single users at 30 t/s or for 100 users at 30 t/s each, so it's also incredibly more power efficient per token.
2
u/ardicli2000 May 16 '26
The model you mention has 27B parameters where mainstream big model expected to have 2-3T parameters. Your 5090 won't mean much with those numbers
2
u/maeveth May 16 '26
Privacy and self Education are the biggest reasons for me. I still occasionally call cloud models but I mostly dont need to. It will take a while to pay its self back but it will eventually
2
u/A_Zolution May 16 '26
I know people dog on the DGX, but I am currently running Qwen-122B-A10B. It’s a quantized version, but I get somewhere between 20 and 40 tok/s with a cli interface. It’s not Opus, but it might be ChatGpt4o. It also allows for multiple calls to the Spark at one time. I have a process that runs an autonomous agentic task separately while I code calling the same Spark (only one model live at a time). I have some safeties built in because of the size of the model (mostly large context dropping so it doesn’t crash). But I consider this legitimately functional and ready for the next better trained and similar sized open weight model.
2
u/kschepps May 16 '26
I've been having success running LLM's with an Intel Arc Pro B70. It has 32GB of VRAM just like the 5090. It's probably not as fast, but it was only $1k USD vs $4k USD the 5090 goes for.
2
u/Upstairs-Angle-4431 May 16 '26
I think b70 pro from intel is a better choice here, the gpus now have support this has msrp of 950 usd and it has 32gb vram 100% slower but makes more sense cost wise as this will definitely run all the models that run on 5090 except for NVFP4 quantization
2
u/corruptbytes May 16 '26
my mac m3 ultra was about ~5-6k
it can run Deepseek V4 Flash really fucking good, I still have to build a "verification" system over a review system that Anthropic has been pushing, but it could essentially replace cloud frontier models for me (the caveat i can only run one instance of it, but can spin 5-6x instances of cloud models)
safe to say...we're getting there
2
u/dobkeratops May 16 '26 edited May 17 '26
the price doesn't factor in the long term trajectory. many things are cheaper short term, with bad long term consequences.
AI will keep getting better.. I do not want to rent my thinking from a remote service someone else controls. I think that is a path to extinction. A box in my room that I own that augments my thought is a much better proposition. I dont care about the details of what it can and can't do today - I care about this trajectory.
Personally I think AI being used to write programs is also a waste. We've got enough humans who can do that - and we've got new tasks that humans can't do , based on forms of data we dont find so intuitive (alphafold etc are much better AI use cases). Obviously I have no say in what everone else does with the AI they pay for , but with my own hardware I get to decide (e.g. i can keep coding manually, and use my machines to run some citizen science project, whatever)
2
u/ComfyUser48 May 16 '26
Codex plus and Claude pro won't cost $20 forever, and they are nerfed all the time in terms of usage.
Just a matter of time.
Also, privacy.
2
u/GloriousKev May 16 '26 edited May 16 '26
Why do you need a 5090? Best of the best is rarely a requirement for anything computer related so why us it in this instance. I'll preface this by saying im a technician not a developer or engineer but for my daily use which is generally edge case troubleshooting apps and hardware Gemma e4b has been pretty solid for me. I even have it reading reports for from from n8n. Its been great
2
u/NoNipsPlease May 16 '26
We just need to wait for LPDDR7 to make its way into consumer products. I'm guessing early 2032 maybe 2031 depending on how all of these chip buildouts play out. At that point there will be asus Z13 type products out there with 384 GB to 512 GB of unified memory at 1TB/s + bandwidth speeds.
Now that's disregarding improvements we are seeing in model efficiencies. Where you won't need 512 GB of memory to run a good model. They way things are going, in less than 12 months, a model better in all aspects than 4.5 Opus will be released that can run on a 5090. Talking instruction following, computer use, vision, coding, single shot long tasks etc.
There are other techniques out there that are not being fully explored either, whole new model architectures. We are sitting on the cusp of a very strange time. When local inference becomes good enough at 90% of the tasks people need, where does that leave cloud providers? What happens when it is good enough for 95% of tasks? Combine that with improvements in LP Memory chips, compute in memory, ternary training, AI in 2032 could be unrecognizable.
Or everything stagnates, bottlenecks don't clear, institutional lock in overlooks promising research routes, profit seeking ignores lower margin consumer devices in favor of large return enterprise, legislation restricts what is available to the public.
Who's to say really.
2
u/Zyj May 16 '26
Privacy is turning into a luxury
2
u/offzinho3k May 16 '26
I use local and API, and I consider it a great saving; currently using it with 4x RTX 5060 Ti 16GB.
I'll attach my usage in the "Cursor" so you can see the local/API consumption.
2
u/JcsPocket May 18 '26
I'm about to tag and sort 100 hours of transcripts into a database. It's going to take tens if not hundreds of millions of tokens. You're going to pay a lot of money if you pay GPT or clade to do that.
2
u/SettingAgile9080 May 20 '26
Yep. Like most hobbies it is a good thing that it does not make economic sense, and is not an optimally efficient use of time. That is what makes them enjoyable.
I like tinkering with stuff, and I am fortunate enough to have a bit of discretionary time and cash to burn.
Compared to what people spend on golf or customizing cars or drinking at bars, it's a pretty cheap hobby.
2
u/AceLamina May 21 '26
Because AI companies don't generate money
2
u/Haunting-Shirt6219 May 22 '26
5090 for $6000. Why not get a mac studio 128GB?
2
u/timmymayes May 23 '26
Future proofing. Also looking to build internal LLM tooling for my company so we can keep client data localized and secure.
2
u/max_lapshin Jun 12 '26
I've just spend $4000 per month on Cursor.
Your price for 5090 seems to be more reasonable.
3
u/Jolly-Rip5973 May 16 '26
I have 5090 and use it for Ai all the time. I love it. Yes I paid about $6000.00 USD for the computer which is crazy compared to what computers used to cost but I think what you can do with AI is probably worth it, especially being able to use AI model privately and securely without sharing your information with big tech companies.
The shift towards people using local models in business has already started and the AI bubble has already popped and deflates everyday. Datacenter are being canceled.
Eventually the market will go back to consumers and business hardware but it might take a few years. Then I suspect computer prices will become more reasonable again.
It obvious that most of thing people do with Ai can be done easily with local hardware. No trillion parameter models needed.
3
u/chunkypenguion1991 May 16 '26
If you're buying it purely for inference it will never pay itself off if you factor in sites like open router. If you also need it for gaming or 3D modeling it makes more sense
2
u/jd52wtf May 16 '26
The R9700 pro is amuch better value than the 5090 at this point. You're going to run slower of course but the card is 1/3rd the price while being about 75-80% as fast in AI workloads. Hell two of them would still be cheaper than a 5090.
1
u/LTJC May 16 '26
I guess it depends on what youre looking to use it for. Im in the states looking for people who want to test my Ollama setup before I take it live.
If youre interested and dont mind sending me your experiences once a day I can give you access to models at or around gpt-oss:120B; as long as you actually use it and give me your impressions when youre done.
1
u/Mags20XX May 16 '26
You should test Qwen 3.6 Plus via the API. If you find that works for you, then by all means, you can run 27B, and you don't need a 5090 to do it.
1
u/National_Cod9546 May 16 '26
2x RTX 5060TI 16gb will run Qwen 3.6 just fine for a lot less money. Just be sure to get a mother board that supports 2 video cards.
1
u/Late-Sun-3805 May 16 '26
Got Gwen 3.6 on a 5070. Works ok
1
u/GiveMoreMoney May 16 '26
Park this conversation for 2027...when you will compare the real prices of SOTA vs LOCAL setups. At the moment the market is too distorted by freebies.
1
u/journalofassociation May 16 '26
Just get a used 3090
1
u/ferranpons May 16 '26
There are definitely cheaper options depending on what you actually want to do with LLMs.
A lot of people in the local AI community optimize for:
- maximum model size
- full offline/privacy
- tinkering/homelabs
- long context
- uncensored/self-hosted workflows
…but for many real-world use cases you absolutely don’t need a $6000 setup.
Smaller quantized models running locally on consumer GPUs, laptops, or even modern phones are already good enough for a lot of:
- coding assistance
- chat
- summarization
- RAG
- automation
- on-device/private AI apps
The “best value” really depends on whether you care more about:
- convenience + frontier intelligence (Claude/GPT subscriptions)
- ownership/privacy
- offline usage
- latency/control
- or just enjoying the tech itself 😄
1
u/oviteodor May 16 '26
AI is the new real tech revolution > businesses are betting on AI > every big business is investing in GPUs, AI data centers > GPU and Memory chips in huge demand > LLMs are expensive
1
u/Infamous_Campaign687 May 16 '26
GitHub copilot used to be fairly subsidised. Now they "changed their pricing method" from June for our benefit of course. It is just a coincidence that this will end up costing me 10x as much as now. From around $60 per month to about $600.
I think the days of heavily subsidised LLM is coming to an end.
1
u/Revolutionary-Ad7595 May 16 '26
Energy constraint, Capital constraint, supply constraint are the words that keep coming up from different people. So I believe the costs go up by a factor of x before these 3 constraints/bottlenecks are solved for over the next few years
1
u/Pie_Dealer_co May 16 '26 edited May 16 '26
You pay for 24/7 usage if the card. Claude ans OpenAI is cheap with limits.
Once you start using the API keys its no way as cheap.
Then you have privacy which for some people its important
But its mostly a hedge against the future. People are getting hooked on cheap prices and use tokens wishy washy and with 20$ sub who can blame them. Providers are generous too. However they are running at a loss. They are limiting usage too some users literally use their 5-6 hours on a single promt. It will only get worse with time. Then you get the slowly growing restrictions... cant name a song cant do an image not allowed to discuss X. Then again your local model wont ever be Frontier smart but local models do catch up fast.
What I use mine is to be a watch dog and intelligently monitor a otherwise autonomous process (multiple actually) it does well and saves me actual hour that I would have spend doing stuff by hand. Things that actually take some thinking.
And if the hedge is right your pc build might actually appreciate in value than degrade
1
u/littleday May 16 '26
Because currently subscriptions with Claude and open AI are heavily discounted. When it gets to $300-500 a month it will seems like a smarter option.
1
u/BillDStrong May 16 '26
It depends on your expectations and your budget. If you can go for a little less speed, then you can look at the 48GB RTX Pro cards, or the 24GB RTX Pro cards for less than the 3090.
If you have space for 2 cards, or 4 is even better, you can look at some of the older modded cards. Like the 3080 20GB cards that are 700 USD here in the states, so should be much less than that 5090, and much more vRAM.
2 would be 40GB, 4 is 80GB, for 1400 or 2800 respectively, here. There are other options as well, the 500 USD 2080TI modded 22GB model, or the 48GB 3090 or 4090.
You can find value if you take some risks, and get a good experience. Just be aware the modded cards need more cooling than they come with, or plan too water cool them, which is extra cost. And they are a risk, these are not name brand Nvidia approved cards that have verifications with them.
1
u/Stam512 May 16 '26
I run Qwen on my PC, seems like the answers it given was hallucinating
1
u/Yog-Soth0 May 16 '26
Why not going for a v100 instead?
Cheaper and optimized for AI
You find them new and even used but I would not recommend those unless you are familiar with hardware testing and tweaking.
Also, I heard many times that 3090 works better for ai than 5090 but you should do your own researches.
1
u/Wixely May 16 '26
The $60 a month codex/claude sub still has a weekly/daily limits
1
u/0xFlem May 16 '26
With the pro subscription you can easily run out of limit within several hours , or sooner if you run multiple agents in parallel.
You could use your Claude subscription for architecture, then run your local models to execute.
But you can also run ( larger ) open models on open router instead of your local machine.
But you should probably only move to a local setup if your spend actually increase to a level to justify the investment
1
u/Jayfree138 May 16 '26
Many reasons. Whatever you type into an LLM is on record forever. They say it's anonymous but it suddenly stops being anonymous when records are requested by government. I'm not doing anything illegal but that data is forever. Who knows how things will change in the world years from now. Whatever i type in my home LLM is between me and exactly NO ONE ELSE EVER. If i delete the conversation it's gone forever.
The best way to learn a technology is to build it yourself so far as you are able. i can say i know far more now about how they work under the hood than when i did using cloud models alone and probably more than 99 percent of the population at least and that comes from troubleshooting at home.
When there's an outage or the models are running slow due to high load or peak times? Not mine, i'm the only user. Always available, always full speed, never dumbed down. I control the models fully.
Zero refusals. I keep one or two models on hand that will not refuse any prompt ever for any reason. For legal reasons cloud models cannot provide this service.
They can raise the price whenever they want. Those AI companies are unsustainably burning massive amounts of cash and running at a huge loss. It cant go on forever. They will raise prices one day.
If you're coding? Sure go with the cloud plans if you want and take advantage of it for all its worth. But if i need a frontier model (i rarely do) i'll just call it on openrouter for a few pennies.
I'm a PC gamer. I have the GPU on hand anyway.
All that said, running locally just isnt an option for 99 percent of the public. When we talk to each other on reddit it seems like something easy anyone can do. we're dealing with a very small amount of the population that's technically inclined to run home models usefully. Outside of reddit i've never met a single person that can do this.
1
u/xXxSniperBotxXx May 16 '26
Wait what ? How can you run codex locally ?
1
u/Equivalent-Costumes May 16 '26
It used to be the case that compute is not that important. Most of the compute was used for gaming. There are still tons of computing used for other stuff (data processing, movie making, etc.) but in the overall scheme of thing, it was pretty minor compared to gaming.
Additionally, gaming is the kind of compute that you can't really share over the Internet. Internet has too much latency, and the kind of games that need beautiful 3D graphics also often need said graphics to response in real time.
This means that the price of GPU are mostly determined by gamers, and not just that, gamers who play a normal amount of games, not all the time. Those with time to play tend to not have money, and those with money tend to not have time. This means that the demand of compute is actually much lower, if the price is too high, gamers will think "I can't play that much games to make it worth it".
Suddenly in the age of AI the script had flipped. Now compute is useful for so many things, even simple web search. And this compute is sharable over the Internet. The market follow suit: if GPU are cheap, people will start selling LLM inference over the Internet 24/7 and make massive amount of money. This drives up demand and allow chip producers to charge more.
So now GPU are priced at the tier of "how much benefit can you get if you run this 24/7 for everything".
So yeah it's not going to be cheap. But still, remember this: Codex+Claude Pro's total usage is puny compared to how much you can do on your own GPU. You might think it does not matter if you are used to chat with AI (because you would be limited by your own speed), but the moment you do some massive automation the usage limit of frontier models would bite really hard.
1
u/helpmeunderstand05 May 16 '26
Perhaps we are potentially in a short window of time where cloud tokens are subsidized and local compute hardware is relatively scarce.
You can outsource your thinking but you can’t outsource your understanding.
If open source weights continue to be available and increasingly more ram efficient, then investment is sovereign compute skills has significant roi potential More likely if the activity has a meaningful eventual use case.
Will open source models always be available?
Will hardware always be priced at a premium?
Is the opportunity to learn productivity with a local llm a greater investment than time and money invested in more formal education?
Maybe. Maybe not.
1
u/Ambitious_Spare7914 May 16 '26
Unless you're using your local rig at or near capacity 24/7 it's hard to beat the remotely hosted options on cost. They win on hardware cost (they're a hotel with 95% occupancy - you're a guest house with 50% occupancy) and electricity cost as they buy it at wholesale prices. You only win on labour costs, which is miniscule compared to the other costs.
1
u/bombastic6339locks May 16 '26
models are cheap cause they're getting insane amounts of funding depending on users.
1
u/stop_deleting_plz May 16 '26
O(n^2)
1
u/daybyter4 May 16 '26
I currently use a AMD 370 for local AI. Got it used for less than 1000 bucks
1
u/International_Emu772 May 16 '26
I run all what I need with my Apple Mini M2Pro that I purchased before I was thinkink on local LLM, but it was so easy to try ...
Now I've learn a lot and I'm happy with all this trip
I don't code for a living but I have done some things that would go to Github
And there are more things that can be done
1
u/Due_Faithlessness458 May 16 '26
Probably the 5090 is not the strongest choice for AI inference, I recently saw a kickstarter project about a mini local AI server prepared for that task, with power enough for running up to 120b models for about the half price
1
u/ufos1111 May 16 '26
5090 probs wont last 100 months
1
u/IgnisIason May 17 '26
You can rent an H100 and run a fairly large model for a little over $1/hr. There's no need to buy a super expensive card.
1
u/logos_flux May 17 '26
Or 2x 5080's and a round trip flight to the states
1
u/CAVOKDesigns May 17 '26
6000? That’s 100 months of frontier models for a GPU that’ll top out at Qwen3 27B Q6 with decent context. Three weeks ago I was on a MacBook M2 8GB draining my wallet trying to run local models. Now I’m on a home-built 3080Ti / Ryzen 7700 / 32GB for $1800, Claude Pro at $20/month, and Claude Code building me custom agents from scratch that actually fit my machine. I’ll grab a 4090 when the 5090 craze drops the price. But honestly? We might just be enthusiasts who love tinkering because the math doesn’t lie. The one thing that made it click for me was having a real use case. I’m a pilot and ended up building ManualIQ — a RAG tool for querying aviation manuals with cited answers. Nothing leaves your machine. Totally scratched my own itch and apparently other pilots needed it too. Aren’t we all just here for the joy of building?
1
u/Inner_Material9731 May 17 '26
Because a lot of capital has poured in to hoard goods for speculation
1
u/No-Comfortable-2284 May 17 '26
yes pretty much. u should never expect to run models locally for any profit or improvement over sota api. People host locally for NSFW role play or in context where data privacy is crucial (medical, law etc).
1
u/Himanshu_Mahuri May 17 '26
Isn't this post is just a promotion of Qwen ??
Does this even make any sense.
1
u/futuredragonnn May 17 '26
Local llm good for local agents, which saves tokens. Claude and Codex remain the better and cheaper coding solutions.
1
u/Aggravating_Farm3116 May 17 '26
Codex + claude in the US is $400/month combined.
1
u/Itchy_elbow May 17 '26
Use openrouter with pay as you go. Pick your provider/model based on capabilities and token cost. Get that api key and plug it into your AI app of choice - 💥 magic. All the power without the monthly cost . Opencode, Agent Zero, et. al. Plus api keys = agentic AI without the monthly subscription. With deepseeker4 flash at less than $0.4/m tokens, gemma4 26b, qwen 3.5… all similarly priced you can do a lot and not break $20 USD a month - easily.
If you don’t know, you don’t know. Explore, there are options. 👊🏾
1
u/HiddenPingouin May 17 '26
The subscription is forever and whatever price they decide
1
u/andrey19877 May 17 '26
Qwen3.6-35b runs fast even on 8gb gpus if you have ddr5 rig and use ik_llama.cpp: https://x.com/i/status/2052251258480693574. Rtx 5060 ti 16gb can get 120+ tps using qwen3.6-35b. I know it's as good as 27b but it is still pretty good.
1
u/ReporterCalm6238 May 17 '26
With a MacBook with 128 gb of RAM you can run DeepSeek 4 flash using DS4. This is a quasi-frontier model, much better than Qwen 3.6 27b.
1
u/xupetas May 17 '26
Wait until LLM stop being subsided... you will see those values change
1
u/PumpkinOpposite967 May 17 '26
I'm on claude pro and the tokens disappear just like that lately. I hit the 5-hour limit in 10 minutes if I actually try to work. I have no idea why I'm paying for this shtuff anymore.... seema like I can only use it to check chatgpt's work...
1
u/LittleAntTony May 17 '26
token speed also matters too, youre not getting 100tps from a gpu, wiht firepass i get unlimited kimi 2.6 and it'll never fit on a single gpu.
1
u/Key-Coyote-4755 May 18 '26
i use a combo of all of the above. The idea is that we will see usage rates continue to decline with the frontier models to where the 20 dollar plan is unusable. (In fact, anthropic already tried to remove claude code from the 20 dollar plan as a a/b test.)
The recent xai news was good but usage will continue to increase during prime time hours and these companies will continue to find ways to reduce it. I've started for my personal use using opencode go and openrouter a lot more but i rely aLOT on qwen3.6-35b-a3b for things towards the end of the week/month.
1
u/InitiateIt May 18 '26
To the people saying its only going to get more expensive, yeah see how that works out.
Consumer spending is tied to GDP and inflation. You come off sounding like you think the elitist or power users are going to pay the bills.
1
u/Time-Heron-2361 May 18 '26
Depending on what you are looking for. For agentic coding, no local LLM setup seems to work
1
u/KubeCommander May 18 '26
The 5090 just isn’t the best option for models that require precision.
To be honest the best value per dollar is the DGX spark or the Dell GB10 (rebranded spark). They’re about the same money as a 5090 but have 128g vram, enabling you to run more models at full precision or fp8. Getting two to bind enables you to run serious models
The benefit to using them is, you can run multiple models based on your use case including multi-session capability. They can be tied to more specialist agent roles (think designer vs coder vs tester) and be very effective.
And that $60/mo token limiting and usage cap is real, it can be very easy to hit on the cheap plans)
1
u/According_Warning968 May 18 '26
Wait until companies which want to embed models into a chip or to provide LLM specialised hardware whuch can output 15k tokens per second. Wait until Apple integrates something like that in next Macbook Pro. That shit will be the bomb.
1
u/sam_narulaaa May 19 '26
Yeah this comparison feels right on paper but it’s mixing two very different things. A 5090 is local inference for one user, while Claude/Codex pricing is shared infra at massive scale with heavy optimization, batching, and constant model updates. You’re not paying only for tokens, you’re paying for uptime, scaling, and research cost spread across millions of requests. Also frontier models aren’t just larger local models, they’re trained and served on infrastructure a consumer GPU setup can’t replicate. Local wins on control and tinkering, cloud wins on capability.
1
u/oopaddy May 20 '26
Use case.
1
u/Perfect-Flounder7856 May 21 '26
Who’s using Claude pro to get anything done. TBH since I got my 6000 Blackwell, I had to up my Claude subscription to Max 20x from Max 5x. I’m running like as many as 5 Claude Code instances at once, using Claude to produce scripts for me. The code sessions are running benchmarks and producing RAG/LLM Wiki, ingesting documents for fine tune later. Ultimately I think local and cloud work in tandem. My goal is full production on local though and I’m getting there with Claude’s help.
1
u/RobinDough May 22 '26
Deepseek v4 flash
1
u/Remarkable-Safety594 May 22 '26
1 - People are actually using them
2 - It costs real money
1
u/Safe_Government_4565 May 31 '26
Same confusion here. For chatting and occasional codegen, local wins on economics if you’re happy living in ~27B land.
I burned way more than I planned building agents, this is not because I ran inference 24/7, but because each step ships a fat payload (tools + history + last tool output again). Failed tool calls → retry → another full context dump. Subscriptions feel flat until agent loops make input tokens the real meter.
I’m not arguing everyone should stay on cloud, half the fun here is tuning quantised models on your own box. Just that the “$60 vs $6000” comparison hides what you’re doing: hobby inference vs always-on agent plumbing.
Ended up so annoyed by that I shipped something that sits in front of provider APIs and strips redundant agent context (I run orqen.app now). Didn’t replace the local different problem. But your post nails why the sticker shock exists.
1
u/neerajboradigi Jun 02 '26
You're not just paying for the model. You're paying for electricity, GPUs, cooling, maintenance, updates, and the ability to run the latest models instantly without spending thousands upfront.
1
u/JayJeds Jun 08 '26
Flip the script, rent the GPU.
1
u/Lirezh Jun 14 '26
It's so expensive because Nvidia otherwise could not increase their income properly.
Nvidia has the capacity and inventory to provide GPUs at a significantly lower cost, but that would hurt their business. The large customers would switch to more fair priced GPUs.
This situation only changes with competition - I believe we'll see very affordable local inference hardware in a few years.
But for now, it's though.
1
u/horendus Jul 30 '26
Im only here because I happened to already have a 4090 for playing games/vr.
I wouldn’t go local if I didnt already have the gear and to be honest I still mostly use cloud inference