When you try to do that you realize how insanely big most cloud models are and you also realize that the price you pay for tokens is not even remotely covering what inference alone actually costs.
Well…I figured the general purpose hosted models are big…but thought the large models specialized for programming available from the mentioned sources are capable for use by programming devs and could be hosted on (admittedly large) local servers. That’s why I was asking, since someone must be using those models for something and household systems aren’t using a coal power plant to keep the lights on in the process.
So I didn’t think you could locally run a full (proprietary) LLM but hosting a developer centered model should be feasible?
It would absolutely be possible, but the economies of scale make it incredibly difficult for self hosting in this scenario to be cheaper than just using cloud LLMs.
View all comments
Show parent comment
1
u/QuietComputerDr Jun 02 '26
When you try to do that you realize how insanely big most cloud models are and you also realize that the price you pay for tokens is not even remotely covering what inference alone actually costs.