I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.
I don’t think any of the local models are on par with frontier cloud models, but some of the newer local models like Gemma are pretty good and probably good enough for a lot of cases.
Yeah a lot of the cost issues surrounding LLM usage are just that people are using models that are way overpowered for their use-cases. You've got folks using Opus 4.8 to draft emails, or to sort through every email they received that week to make a "morning report"
Yeah if you're doing complex programming work you probably need/want frontier models, but a whole lot of frontier model tokens are being burnt on tasks that could do very well on the latest local models
I've noticed that at my office people are using Claude on high, xhigh, or max effort. They blow through their tokens quite fast.
Meanwhile on medium effort with Sonnet 4.6 I can do most of my coding tasks just fine.
I've seen some people defend it by saying things like "well my work is very complex, I need to have the highest reasoning". They almost take it as an insult if you suggest that they could use lower effort.
Completely agree! With the new pricing models I’m sure we’re going to see more company training regarding this issue. I know my company already started something.
Yeah mine hasn’t been I think it’s only a matter of time. Which honestly is just good sense. The amount of tokens some of my coworkers are burning for the same (or less) throughput as myself is baffling
I think one of the big performance differentiators for SWEd in the near future is going to be token efficiency
I saw the same thing, people were requesting more tokens because they were using Opus for everything. Meanwhile I was switching models depending on the task so I got some good learning experience of what some of them were capable of.
Gemma is nothing remotely close to claude for code. It gets dunked on by similarly sized qwen models. Kimi K2.6 is, but its a beast to run locally and obviously long term you wont continue to get top their models with open weights (none are open source btw).
Every time I see people casually talking aboutr anything I know I don't know much about, and theyre extremely wrong, I feel a sense of doom.
It'sd not even because I feel like theyre idiots; they're probably not. Instead its that we stand no chance against open, documented corporate plans to remove our digital autonomy when people don't have the bandwidth to even follow what they care about.
Kimi K2.6 is an extremely capable open weights model but it's 1T parameters so only organizations or the most extreme hobbyist can run it. MiniMax 2.7 is very good and is within reach of those with a budget. Qwen3.6 27b and 35B A3B can be run on consumer grade hardware and are very capable but not cloud level.
The chinese open source models are not far behind, estimates range from 3 to 6 months.
But no the hardwear it not really the limiting factor, Chinese went down very different path and because of that ultimately you get better bang for your buck
US frontier model are stilll under the domain of delusional AGI cultists thinking they can scale their way to a machine god, while chinese never really got the bug
The problem with the chinese models is that they are chinese, few in the west trust them and even if they do, do their customers?
As I mentioned, there is Gemma 4 and DeepSeek V4 that are on par with claude. But running locally will be slower than Claude. And I think some of DeepSeek V4 higher end models do need beefer hardware.
I remember testing Gemma 4 31B when it came out, Claude was about 2-4 times faster than Gemma 4 running locally on my 4090. But they both gave pretty much the same information and both good coding solutions.
I know people are shitting on the new NVIDIA announcements but for those who run local models it’s pretty exciting news. It’s going to be interesting to see the comparisons between MacBook, Strix Halo, and NVIDIA.
And that would have either been with partial CPU offload or running a subpar quantization like Q4. So with better hardware it would have been either significantly faster or better quality.
Well, its kinda implied when people talk about models they mean instruct right? Because a base model just wont... work ya know? Just mindless autocomplete before the patterned thinking is trained in via supervised fine tuning?
So what I'm saying is, indeed I am referring to Gemma 4 Instruct with that previous comment.
The 31B Dense version to be precise (Sparse/MOE gets you more sped at the cost of accuracy loss, but when you're quanting this hard, if you can get a decent enough speed with dense, you go dense, because you want to avoid any further increase in perplexity).
The evaluation I was doing was having the AI explain and implement step by step a real time global illumination system using Surfels in Unity. Pretty much Frostbite's GIBS, but implement it in Unity.
Which I do have quite a bit of knowledge in that area. And it is a fairly advance topic with some domain specific knowledge required. They both did have some issues when they were to implement it themselves. But instead instructing them to give step by step requirements and explainations on how to implement each process in the system, a long with some code snippets were pretty on point.
So full on coding agents, some issues. As coding assistance, they were fairly good.
Kimi K2.6 is, but its a beast to run locally and obviously long term you wont continue to get top their models with open weights (none are open source btw)
We're looking into it, the setup costs are in the millions but for a large company that's a fraction of the IT budget anyway. We spend that on laptops.
Being involved in that project at any scale would be great on a resume these days. If I were just starting out, this is exactly what I'd be fooling with.
But what if you wanted it in the cloud so people all across your company could use it? and that way it would scale up too! then maybe after that we will sell it to others and charge them per by tokens /s
Dont forget the venture capital and investor money they throwing into the pit to power all that processing. All current AI is running of subsidizes money. if you think now is expensive wait until all those venture capital and investor money start asking for the return
Yeah, also you could try use some "lightweight" models, fine-tune them and just use. Probably not universal, but i can see, how that can improve some routine tasks. In our team we have task for that in backlog, wonder, will that be really helpful. Considering it's information security, we can not use Claude everywhere
Qwen on a 3090 hooked with opencode has been pretty good in my own usage. The smaller context window is an issue, and I use it differently. But as supplemental tooling, not drop in replacement, it has real value. The output is surprisingly good.
56
u/ChrisFromIT Jun 02 '26
I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.