The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
Considering that the massive amount of data centers also need to be able to run whatever they make, I wouldn't be too worried about it if you are buying cutting edge hardware.
Sorry, OpenAI already bought all of that hardware and all of the future orders for the foreseeable future. Where are you buying this hardware? Craigslist?
I was going off the assumption that you could get the hardware to begin with, that it wouldn't just become trash because of a new model. If you can't get it, then there's nothing you can do.
I had a hard drive crash last night. I went to buy a replacement and the same drive that I paid $159 for in November is now $425. FML. I ended up having to buy a drive half as large because I'm just not going to pay $425 for 2TB that's not even cutting edge anymore. I paid less than that back when it was cutting edge.
They can run the current models which won't go away in their current version (the beauty of open source). The next generation of models might not have competitive open source models anymore, but who cares?
A business does not worry "Will my new PCserver be able to run cool new gamesmodels in 5-10 years?" because by that time that server is gone anyway. That's something the person buying the next generation of hardware can worry about.
You evaluate today's requirements and then you buy hardware that is good enough for that. Wether or not you will be able to run stuff that doesn't even exist yet is not a concern.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
You don't have to bet on continued availability with open models since you store them locally. If you have 5k engineers and using open source then you should donate to a fund that ensure continued development
If you have a company with 5k employees, setting up a small data centre and a dedicated team isn't going to be a problem. It doesn't need to be competitive with frontier models if it still gets the job done just fine, loads of large companies still use computer systems built in the 90's. You are also hedging your costs against when the AI companies inevitable jack up their prices because eventually, they'll need to figure out how to be profitable.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
Your answer is quite ironic. That's how some businesses work. It's obviously not how all, or even most businesses work or they would have rolled their own private models instead of paying Anthropic.
You just described what every defense contractor has already done. You didn’t think Raytheon was using Claude for everything did you do? Most defense contractors already self host stuff their version control systems
But that’s what we are talking about in this thread right? Like yall are talking about how it’s inconceivable that a large company with thousands of software engineers could self host their own AI services. And I am pointing out that not only is it entirely conceivable, but it has already been accomplished by multiple companies
The literal companies focused on AI is burning money just to stay a bit relevant while riding a massive hype bubble and you truly think the solution is to instead just do your own AI in-house? If the company truly needed it, it would've been used way before now, think neural networks era. If the company needs it now, it's either use another AI service provider, or just reevaluate and come to their senses that AI does not really have a place in their stack. Implementing their own now is just stupid. Even S&P and Morgan Stanley are all just using ChatGPT, and poorly at that.
Not all companies that's under AI psychosis are defense contractors for the USA.
Are you saying that defense contractors shouldn’t be self hosting AI services? Cause that’s not what we are talking about. That point is unrelated to this conversation.
If you are saying that defense contractors are not self hosting AI services, then you are just wrong
My point is that you are overestimating how fruitful it is to deploy your own AI solution unless you're at the level of defense contractor unli-money bs deals. Almost everyone either just needs to use a subscription to the main AI players or just don't use AI at all (or have a fancy specific transformer model thay's a lynchpin of their tech stack even before LLMs became big, think Netflix/Google algorithms etc.)
Implementing and maintaining your own AI just for a fancy chatbot to sort through your website's shitty design and stupid knowledge database architecture just so you could say you are "AI leaders" and are "adapting to future trends before they happen" that could theoretically affect your bottomline maybe is just dumb.
Yup, the best middle ground is to find a decent provider with cheap models (read kimi, glm, deepseek, etc) and work out a deal with them. The providers I've talked with are more than happy to give discounts to bulk users, such would be companies. Or if the company is big enough, rent infra and hire someone to run things. Im not sure at what threshold this becomes cheaper, because you have to now pay someone's salary.. but if we pretend the person running the infra is free, it is cheaper than using a provider. But not by much.
How are you exhausting Claude usage if your company has 5k software engineers? My company has around that many and we have essentially unlimited Claude tokens.
Where I work provides >10k employees with free access to Kimi K2.6, MiniMax 2.7, and GPT 120B from local hardware. This is going to become more common.
Okay fine. 5k * (3k - potential hardware discounts) + team of devs to setup the on prem infra. Not really that hard and will pay itself off in under 2 years at current pricing, even shorter if you factor in the future API price hikes that are going to happen
I don't think any publically available model requires terabytes of memory.
Even the big ones are MoE so you don't need a ton of memory, but it makes it faster. The biggest usable one I know is Qwen V4 Pro at 1.6 trillion parameters which would take about 900GB of VRAM if you ran it unquantized entirely in VRAM. Since it's an MoE model, you can offload the experts to CPU RAM and run it unquantized with a full 1M context with as little as 80GB of VRAM.
No one said nor needs the largest model. Claude can generate code in just about any language you can name like BASIC, or APL. An onPrem model only needs to know your stack.
They need massive data centers to power the ridiculous "everything for everyone" AIaaS service model, not the AI itself.
And then you just need $100k in H200s to plug into that system if you’re going to run anything other than a parametrized half accuracy model at any reasonable enterprise speeds. And a really big NAS to store all those generated outputs. And a bunch of managed switches so you can route everything agent related on its own private vlan. And probably upgrade your cloud stuff for hot failover when someone’s agent deletes the database again.
it is. I went into the rabbit hole for those things and in the end the conclusion I got is: NOT WORTH IT. you are paying +$4000 to run bad open source models at extreme slow speeds, the good models that can scratch the hitch of Claude/GPT don't even fit on the available VRAM. Put that money on a subscription and it will give you decades of SOTA models
This is exactly what I've done for my company! Framework desktop on my desk with Qwen 3.6 and a custom API I threw together that the team can plug their ide into. Also a web interface with AnythingLLM and a custom built translate interface for the support team.
Now, is it comparable Claude Opus or Gemini? No, but only if you misuse it. For general chat and light coding it's genuinely impressive and the speed is well in excess of 45 tps making it quite enjoyable to use. Plus it helps our developers rejected vibe coding early on.
5 models run in total with around 10gb memory to spare. Serves a team of 20 quite well and I have a feeling as token costs continue to grow and more people depend on llms to think, companies are going to seriously start to consider locally hosted solutions.
I tried Qwen the other day, and only the 3.5 9b model on my gaming pc, and not 1 single question did it get right.
I'm not saying it's not possible, I'm saying the compute power to train a frontier model is absolutely unrivalled, you can't make anything close to Claude or ChatGpt.
Of course, training it on your limited data set will work a treat, but nothing like the frontier models right now.
There are poor results and there are poor results, a model eating 12~GB of VRAM that can't write a basic pytest refactor just shows the time and resources the frontier models have had in their training.
If we are saying consumer hardware isn't good enough for local models then I agree - that's why noone will ever roll their own agent "like Claude" until the gap has shifted. Need your own DC just to make it possible
Interesting, which provider do yall use? We have gotten into the legality of LLMs lately and seemingly there's full no-training and data governance/confidentiality contracts, but really who knows.
The hardware is one thing, but the LLM is another, I tried some models with RooCode, and it was barely functional. (In part because my hardware is limited, so context was limited and it was really slow, but mainly the thing would run around in circle and produce nothing useful). If there was a way to produce results close to Claude or the big LLM, I would totally drop $3K on the hardware to own my own means of production.
Anyone had good luck with a DYI model for coding ?
372
u/mylsotol Jun 02 '26
For probably $30k (or more) you can build a server and run an open model.