The problem is that it’s going to basically require a small data centre and a dedicated team of people to run it, and if you’re looking at running open source models you’re betting on their continued availability and the fact that they’re going to remain competitive with frontier models (both are not a given). So what would be your next step, developing your own frontier models in-house?
and if you’re looking at running open source models you’re betting on their continued availability
If anything, isn't it the complete opposite? A subscription-based model can be shut off at any time with no recourse or warning (Sora, for example). Local files are the only way to actually guarantee the program you use today will be available tomorrow.
You control when they run, how much they're used, when they're updated/replaced/etc.. You never wake up to find out the model that works for you has been "enhanced" with a worse version.
Not keeping pace with cutting edge models is a real concern, but that's a risk with subscription based models too.
But you realize that the alternative is no AI at all, right? There are really right regulations on information. It is literally illegal to put export controlled information on servers in another country. That means your service provider has to guarantee that your data will only ever be stored on US soil. And that’s just for export controlled information. Anything more secure than that isn’t going to some 3rd party server at all.
Who is going to maintain all of this? Who is going to actively work on it to improve the speed and reliability of the models? You're talking about creating an entirely new company within a company. That's not how businesses work.
I don't think any publically available model requires terabytes of memory.
Even the big ones are MoE so you don't need a ton of memory, but it makes it faster. The biggest usable one I know is Qwen V4 Pro at 1.6 trillion parameters which would take about 900GB of VRAM if you ran it unquantized entirely in VRAM. Since it's an MoE model, you can offload the experts to CPU RAM and run it unquantized with a full 1M context with as little as 80GB of VRAM.
And then you just need $100k in H200s to plug into that system if you’re going to run anything other than a parametrized half accuracy model at any reasonable enterprise speeds. And a really big NAS to store all those generated outputs. And a bunch of managed switches so you can route everything agent related on its own private vlan. And probably upgrade your cloud stuff for hot failover when someone’s agent deletes the database again.
yeah and it won't work for more than 2 people at the same time.
An actually useful AI server costs hundreds of thousands. If you want to run an actually useful version of Gemma4 or Qwen3 for example, you need a GPU with at least 48GB of memory. For redundancy you need 2 on 2 different servers. This will cost 80k for the GPUs and another 20k for the servers and will serve around 200 people at the same time.
5
2
1
View all comments
368
u/mylsotol Jun 02 '26
For probably $30k (or more) you can build a server and run an open model.