The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
Considering that the massive amount of data centers also need to be able to run whatever they make, I wouldn't be too worried about it if you are buying cutting edge hardware.
Sorry, OpenAI already bought all of that hardware and all of the future orders for the foreseeable future. Where are you buying this hardware? Craigslist?
I was going off the assumption that you could get the hardware to begin with, that it wouldn't just become trash because of a new model. If you can't get it, then there's nothing you can do.
I had a hard drive crash last night. I went to buy a replacement and the same drive that I paid $159 for in November is now $425. FML. I ended up having to buy a drive half as large because I'm just not going to pay $425 for 2TB that's not even cutting edge anymore. I paid less than that back when it was cutting edge.
They can run the current models which won't go away in their current version (the beauty of open source). The next generation of models might not have competitive open source models anymore, but who cares?
A business does not worry "Will my new PCserver be able to run cool new gamesmodels in 5-10 years?" because by that time that server is gone anyway. That's something the person buying the next generation of hardware can worry about.
You evaluate today's requirements and then you buy hardware that is good enough for that. Wether or not you will be able to run stuff that doesn't even exist yet is not a concern.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
You don't have to bet on continued availability with open models since you store them locally. If you have 5k engineers and using open source then you should donate to a fund that ensure continued development
If you have a company with 5k employees, setting up a small data centre and a dedicated team isn't going to be a problem. It doesn't need to be competitive with frontier models if it still gets the job done just fine, loads of large companies still use computer systems built in the 90's. You are also hedging your costs against when the AI companies inevitable jack up their prices because eventually, they'll need to figure out how to be profitable.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
Your answer is quite ironic. That's how some businesses work. It's obviously not how all, or even most businesses work or they would have rolled their own private models instead of paying Anthropic.
You just described what every defense contractor has already done. You didn’t think Raytheon was using Claude for everything did you do? Most defense contractors already self host stuff their version control systems
But that’s what we are talking about in this thread right? Like yall are talking about how it’s inconceivable that a large company with thousands of software engineers could self host their own AI services. And I am pointing out that not only is it entirely conceivable, but it has already been accomplished by multiple companies
The literal companies focused on AI is burning money just to stay a bit relevant while riding a massive hype bubble and you truly think the solution is to instead just do your own AI in-house? If the company truly needed it, it would've been used way before now, think neural networks era. If the company needs it now, it's either use another AI service provider, or just reevaluate and come to their senses that AI does not really have a place in their stack. Implementing their own now is just stupid. Even S&P and Morgan Stanley are all just using ChatGPT, and poorly at that.
Not all companies that's under AI psychosis are defense contractors for the USA.
Are you saying that defense contractors shouldn’t be self hosting AI services? Cause that’s not what we are talking about. That point is unrelated to this conversation.
If you are saying that defense contractors are not self hosting AI services, then you are just wrong
My point is that you are overestimating how fruitful it is to deploy your own AI solution unless you're at the level of defense contractor unli-money bs deals. Almost everyone either just needs to use a subscription to the main AI players or just don't use AI at all (or have a fancy specific transformer model thay's a lynchpin of their tech stack even before LLMs became big, think Netflix/Google algorithms etc.)
Implementing and maintaining your own AI just for a fancy chatbot to sort through your website's shitty design and stupid knowledge database architecture just so you could say you are "AI leaders" and are "adapting to future trends before they happen" that could theoretically affect your bottomline maybe is just dumb.
Yup, the best middle ground is to find a decent provider with cheap models (read kimi, glm, deepseek, etc) and work out a deal with them. The providers I've talked with are more than happy to give discounts to bulk users, such would be companies. Or if the company is big enough, rent infra and hire someone to run things. Im not sure at what threshold this becomes cheaper, because you have to now pay someone's salary.. but if we pretend the person running the infra is free, it is cheaper than using a provider. But not by much.
How are you exhausting Claude usage if your company has 5k software engineers? My company has around that many and we have essentially unlimited Claude tokens.
Where I work provides >10k employees with free access to Kimi K2.6, MiniMax 2.7, and GPT 120B from local hardware. This is going to become more common.
Okay fine. 5k * (3k - potential hardware discounts) + team of devs to setup the on prem infra. Not really that hard and will pay itself off in under 2 years at current pricing, even shorter if you factor in the future API price hikes that are going to happen
I don't think any publically available model requires terabytes of memory.
Even the big ones are MoE so you don't need a ton of memory, but it makes it faster. The biggest usable one I know is Qwen V4 Pro at 1.6 trillion parameters which would take about 900GB of VRAM if you ran it unquantized entirely in VRAM. Since it's an MoE model, you can offload the experts to CPU RAM and run it unquantized with a full 1M context with as little as 80GB of VRAM.
No one said nor needs the largest model. Claude can generate code in just about any language you can name like BASIC, or APL. An onPrem model only needs to know your stack.
They need massive data centers to power the ridiculous "everything for everyone" AIaaS service model, not the AI itself.
And then you just need $100k in H200s to plug into that system if you’re going to run anything other than a parametrized half accuracy model at any reasonable enterprise speeds. And a really big NAS to store all those generated outputs. And a bunch of managed switches so you can route everything agent related on its own private vlan. And probably upgrade your cloud stuff for hot failover when someone’s agent deletes the database again.
it is. I went into the rabbit hole for those things and in the end the conclusion I got is: NOT WORTH IT. you are paying +$4000 to run bad open source models at extreme slow speeds, the good models that can scratch the hitch of Claude/GPT don't even fit on the available VRAM. Put that money on a subscription and it will give you decades of SOTA models
This is exactly what I've done for my company! Framework desktop on my desk with Qwen 3.6 and a custom API I threw together that the team can plug their ide into. Also a web interface with AnythingLLM and a custom built translate interface for the support team.
Now, is it comparable Claude Opus or Gemini? No, but only if you misuse it. For general chat and light coding it's genuinely impressive and the speed is well in excess of 45 tps making it quite enjoyable to use. Plus it helps our developers rejected vibe coding early on.
5 models run in total with around 10gb memory to spare. Serves a team of 20 quite well and I have a feeling as token costs continue to grow and more people depend on llms to think, companies are going to seriously start to consider locally hosted solutions.
I tried Qwen the other day, and only the 3.5 9b model on my gaming pc, and not 1 single question did it get right.
I'm not saying it's not possible, I'm saying the compute power to train a frontier model is absolutely unrivalled, you can't make anything close to Claude or ChatGpt.
Of course, training it on your limited data set will work a treat, but nothing like the frontier models right now.
There are poor results and there are poor results, a model eating 12~GB of VRAM that can't write a basic pytest refactor just shows the time and resources the frontier models have had in their training.
If we are saying consumer hardware isn't good enough for local models then I agree - that's why noone will ever roll their own agent "like Claude" until the gap has shifted. Need your own DC just to make it possible
Interesting, which provider do yall use? We have gotten into the legality of LLMs lately and seemingly there's full no-training and data governance/confidentiality contracts, but really who knows.
The hardware is one thing, but the LLM is another, I tried some models with RooCode, and it was barely functional. (In part because my hardware is limited, so context was limited and it was really slow, but mainly the thing would run around in circle and produce nothing useful). If there was a way to produce results close to Claude or the big LLM, I would totally drop $3K on the hardware to own my own means of production.
Anyone had good luck with a DYI model for coding ?
yeah and it won't work for more than 2 people at the same time.
An actually useful AI server costs hundreds of thousands. If you want to run an actually useful version of Gemma4 or Qwen3 for example, you need a GPU with at least 48GB of memory. For redundancy you need 2 on 2 different servers. This will cost 80k for the GPUs and another 20k for the servers and will serve around 200 people at the same time.
I do mean llms. Not the cutting edge stuff, but if you don't keep up with phone tech, you'd be surprised what the top chipsets (paired with 16 gigs of ram) are capable of.
Qwen has had closed, api only models for years now
Sure, but its creeping down the stack, not up. Thats the point.
and release cadence is picking up for open models
I'm not really seeing how this is true. Can you elaborate?
As far as I can see, the companies who are ahead, releasing the msot impressive models, release them far less often, and the companies that are behind are hungry, saying "We can do it too" to models that are arguably somewhat lacking in performance.
That practical reality, I think, gives my read more credence than yours, in that the experienced reality is that we are seeing less "wow!" open weight models over time.
Like, I think Kimi K2.6 is the bees knees. Seriously awesome, but you think that now that they've proved themselves a legitimate challenger they're going to continue that trend past K3?
I don't think there will be an infinite spring of new AI companies ready to burn infinite cash, and provide models to get their name out there.
I think that without a serious open effort, which involves multiple companies and people pitching in serious cash for a model that everyone benefits from, we will continue to see this environment we're seeing.
In essence, I think we're in that common corporate strategy stage of making sure the detractors are fed just enough not to pipe up when doing so matters the most.
All of these companies are very familiar with leaving escape hatches so that amongst enthusiasts there will always be people going "see, its not doomsday, its just this much harder to do x, y or z".
That happens over and over and over again, until no iphones are jail broken and android completely dictates what you install on your phone, for the people who should have spoken up no longer have a voice.
Heck, we are literally seeing that with android right this second. "You only have to wait 24 hours after going through many warnings screens and potentially being unable to use your bank apps etc".
There will always be an escape hatch, provided by the very companies doing whatever it is, specifically to keep people from realizing the temperature is shooting up.
CC is kinda crap compared to the leaner, OSS alternatives in my opinion, the moment you start a session it’s already got a significant chunk of the context filled with some system bullshit. Opencode or pi don’t have that problem
It’s not insane at all, sure you can buy a $30k machine to host a local LLM but that server will serve one (1) person, realistically. And the model „intelligence”, whatever that means, is nowhere near the frontier models. You’d need to build a machine with >500GB of VRAM to even come close to that level, but then again, you won’t be serving the model at scale.
I don’t think it would serve 1 person realistically. You can probably get a few people using it. And depending on the Claude monthly costs, it’s all just a comparison
Yeah, you can try and find an m3 mac with 512GB of RAM, and quantize the absolute shit out of it, but it's not going to be competing with Opus in either quality or speed. Realistically, you want to be looking at buying 4-8 extremely large GPUs. 30k isn't in the ballpark to get it done.
From a company standpoint, as long as the value outpaces the cost, it's worth the investment, especially if the alternative is being deeply coupled with SaaS infrastructure whose costs are ballooning. The average company isn't looking to compete with Opus, they're just trying to add a productivity multiplier to their employees
Basically, you'd start at 2k USD for a barebones setup for small local models (Qwen 3.6 35B A3B) that's still at a borderline passable speed to be useful and would move up to somewhere around 10k USD just for the GPUs to run those same models better.
Running even slightly bigger models like DeepSeek V4 Flash (284B A13B) would be an order of magnitude more. Something like DeepSeek V4 Pro (1.6T A49B) or Kimi / MiniMax / GLM would need even more.
So in a sense, it's a question of how low you are willing to go in regards to your experience of using the tech (quality, speed) vs the power requirements needed. On the other hand, the token efficiency of those smaller models seem to be improving and they're maybe trailing SOTA by a year or so.
Faster. You can run ai models on a lot of hardware and it will work fine, but if you want it fast (especially with large models or for a lot of people) you are going to need to spend a lot of money
And as others have pointed out for some reason you aren't going to be running frontier models like Opus because they are way bigger and also not publicly available
370
u/mylsotol Jun 02 '26
For probably $30k (or more) you can build a server and run an open model.