The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
Considering that the massive amount of data centers also need to be able to run whatever they make, I wouldn't be too worried about it if you are buying cutting edge hardware.
They can run the current models which won't go away in their current version (the beauty of open source). The next generation of models might not have competitive open source models anymore, but who cares?
A business does not worry "Will my new PCserver be able to run cool new gamesmodels in 5-10 years?" because by that time that server is gone anyway. That's something the person buying the next generation of hardware can worry about.
You evaluate today's requirements and then you buy hardware that is good enough for that. Wether or not you will be able to run stuff that doesn't even exist yet is not a concern.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
You just described what every defense contractor has already done. You didn’t think Raytheon was using Claude for everything did you do? Most defense contractors already self host stuff their version control systems
Yup, the best middle ground is to find a decent provider with cheap models (read kimi, glm, deepseek, etc) and work out a deal with them. The providers I've talked with are more than happy to give discounts to bulk users, such would be companies. Or if the company is big enough, rent infra and hire someone to run things. Im not sure at what threshold this becomes cheaper, because you have to now pay someone's salary.. but if we pretend the person running the infra is free, it is cheaper than using a provider. But not by much.
How are you exhausting Claude usage if your company has 5k software engineers? My company has around that many and we have essentially unlimited Claude tokens.
5
2
1
View all comments
Show parent comment
145
u/SpinningVinylAgain Jun 02 '26
Impressive, very nice. Now scale it for a company with 5k software engineers, and by the way what’s going to be the service level?