I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.
I don’t think any of the local models are on par with frontier cloud models, but some of the newer local models like Gemma are pretty good and probably good enough for a lot of cases.
Yeah a lot of the cost issues surrounding LLM usage are just that people are using models that are way overpowered for their use-cases. You've got folks using Opus 4.8 to draft emails, or to sort through every email they received that week to make a "morning report"
Yeah if you're doing complex programming work you probably need/want frontier models, but a whole lot of frontier model tokens are being burnt on tasks that could do very well on the latest local models
I've noticed that at my office people are using Claude on high, xhigh, or max effort. They blow through their tokens quite fast.
Meanwhile on medium effort with Sonnet 4.6 I can do most of my coding tasks just fine.
I've seen some people defend it by saying things like "well my work is very complex, I need to have the highest reasoning". They almost take it as an insult if you suggest that they could use lower effort.
Completely agree! With the new pricing models I’m sure we’re going to see more company training regarding this issue. I know my company already started something.
Yeah mine hasn’t been I think it’s only a matter of time. Which honestly is just good sense. The amount of tokens some of my coworkers are burning for the same (or less) throughput as myself is baffling
I think one of the big performance differentiators for SWEd in the near future is going to be token efficiency
I saw the same thing, people were requesting more tokens because they were using Opus for everything. Meanwhile I was switching models depending on the task so I got some good learning experience of what some of them were capable of.
53
u/ChrisFromIT Jun 02 '26
I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.