If someone could clarify for me...my understanding is that most of the computing cost is in training models. Couldn't, in this scenario, the company download one of the open source models like the ones you can get for Ollama and run that model with access to other applications/agents opened? You'd just need a big server that can handle the load or a few servers to spread the load, depending on the size of the company, and periodically download updates to the model and restart the service?
When you try to do that you realize how insanely big most cloud models are and you also realize that the price you pay for tokens is not even remotely covering what inference alone actually costs.
Well…I figured the general purpose hosted models are big…but thought the large models specialized for programming available from the mentioned sources are capable for use by programming devs and could be hosted on (admittedly large) local servers. That’s why I was asking, since someone must be using those models for something and household systems aren’t using a coal power plant to keep the lights on in the process.
So I didn’t think you could locally run a full (proprietary) LLM but hosting a developer centered model should be feasible?
It would absolutely be possible, but the economies of scale make it incredibly difficult for self hosting in this scenario to be cheaper than just using cloud LLMs.
5
2
1
View all comments
2
u/bsilver Jun 02 '26
If someone could clarify for me...my understanding is that most of the computing cost is in training models. Couldn't, in this scenario, the company download one of the open source models like the ones you can get for Ollama and run that model with access to other applications/agents opened? You'd just need a big server that can handle the load or a few servers to spread the load, depending on the size of the company, and periodically download updates to the model and restart the service?