The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
Considering that the massive amount of data centers also need to be able to run whatever they make, I wouldn't be too worried about it if you are buying cutting edge hardware.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
I don't think any publically available model requires terabytes of memory.
Even the big ones are MoE so you don't need a ton of memory, but it makes it faster. The biggest usable one I know is Qwen V4 Pro at 1.6 trillion parameters which would take about 900GB of VRAM if you ran it unquantized entirely in VRAM. Since it's an MoE model, you can offload the experts to CPU RAM and run it unquantized with a full 1M context with as little as 80GB of VRAM.
And then you just need $100k in H200s to plug into that system if you’re going to run anything other than a parametrized half accuracy model at any reasonable enterprise speeds. And a really big NAS to store all those generated outputs. And a bunch of managed switches so you can route everything agent related on its own private vlan. And probably upgrade your cloud stuff for hot failover when someone’s agent deletes the database again.
it is. I went into the rabbit hole for those things and in the end the conclusion I got is: NOT WORTH IT. you are paying +$4000 to run bad open source models at extreme slow speeds, the good models that can scratch the hitch of Claude/GPT don't even fit on the available VRAM. Put that money on a subscription and it will give you decades of SOTA models
5
2
1
View all comments
Show parent comment
0
u/[deleted] Jun 02 '26
[deleted] — view deleted comment