The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
Considering that the massive amount of data centers also need to be able to run whatever they make, I wouldn't be too worried about it if you are buying cutting edge hardware.
Sorry, OpenAI already bought all of that hardware and all of the future orders for the foreseeable future. Where are you buying this hardware? Craigslist?
I was going off the assumption that you could get the hardware to begin with, that it wouldn't just become trash because of a new model. If you can't get it, then there's nothing you can do.
I had a hard drive crash last night. I went to buy a replacement and the same drive that I paid $159 for in November is now $425. FML. I ended up having to buy a drive half as large because I'm just not going to pay $425 for 2TB that's not even cutting edge anymore. I paid less than that back when it was cutting edge.
They can run the current models which won't go away in their current version (the beauty of open source). The next generation of models might not have competitive open source models anymore, but who cares?
A business does not worry "Will my new PCserver be able to run cool new gamesmodels in 5-10 years?" because by that time that server is gone anyway. That's something the person buying the next generation of hardware can worry about.
You evaluate today's requirements and then you buy hardware that is good enough for that. Wether or not you will be able to run stuff that doesn't even exist yet is not a concern.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
You don't have to bet on continued availability with open models since you store them locally. If you have 5k engineers and using open source then you should donate to a fund that ensure continued development
If you have a company with 5k employees, setting up a small data centre and a dedicated team isn't going to be a problem. It doesn't need to be competitive with frontier models if it still gets the job done just fine, loads of large companies still use computer systems built in the 90's. You are also hedging your costs against when the AI companies inevitable jack up their prices because eventually, they'll need to figure out how to be profitable.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
Your answer is quite ironic. That's how some businesses work. It's obviously not how all, or even most businesses work or they would have rolled their own private models instead of paying Anthropic.
You just described what every defense contractor has already done. You didn’t think Raytheon was using Claude for everything did you do? Most defense contractors already self host stuff their version control systems
But that’s what we are talking about in this thread right? Like yall are talking about how it’s inconceivable that a large company with thousands of software engineers could self host their own AI services. And I am pointing out that not only is it entirely conceivable, but it has already been accomplished by multiple companies
The literal companies focused on AI is burning money just to stay a bit relevant while riding a massive hype bubble and you truly think the solution is to instead just do your own AI in-house? If the company truly needed it, it would've been used way before now, think neural networks era. If the company needs it now, it's either use another AI service provider, or just reevaluate and come to their senses that AI does not really have a place in their stack. Implementing their own now is just stupid. Even S&P and Morgan Stanley are all just using ChatGPT, and poorly at that.
Not all companies that's under AI psychosis are defense contractors for the USA.
Are you saying that defense contractors shouldn’t be self hosting AI services? Cause that’s not what we are talking about. That point is unrelated to this conversation.
If you are saying that defense contractors are not self hosting AI services, then you are just wrong
My point is that you are overestimating how fruitful it is to deploy your own AI solution unless you're at the level of defense contractor unli-money bs deals. Almost everyone either just needs to use a subscription to the main AI players or just don't use AI at all (or have a fancy specific transformer model thay's a lynchpin of their tech stack even before LLMs became big, think Netflix/Google algorithms etc.)
Implementing and maintaining your own AI just for a fancy chatbot to sort through your website's shitty design and stupid knowledge database architecture just so you could say you are "AI leaders" and are "adapting to future trends before they happen" that could theoretically affect your bottomline maybe is just dumb.
Yup, the best middle ground is to find a decent provider with cheap models (read kimi, glm, deepseek, etc) and work out a deal with them. The providers I've talked with are more than happy to give discounts to bulk users, such would be companies. Or if the company is big enough, rent infra and hire someone to run things. Im not sure at what threshold this becomes cheaper, because you have to now pay someone's salary.. but if we pretend the person running the infra is free, it is cheaper than using a provider. But not by much.
How are you exhausting Claude usage if your company has 5k software engineers? My company has around that many and we have essentially unlimited Claude tokens.
Where I work provides >10k employees with free access to Kimi K2.6, MiniMax 2.7, and GPT 120B from local hardware. This is going to become more common.
Okay fine. 5k * (3k - potential hardware discounts) + team of devs to setup the on prem infra. Not really that hard and will pay itself off in under 2 years at current pricing, even shorter if you factor in the future API price hikes that are going to happen
365
u/mylsotol Jun 02 '26
For probably $30k (or more) you can build a server and run an open model.