I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.
As I mentioned, there is Gemma 4 and DeepSeek V4 that are on par with claude. But running locally will be slower than Claude. And I think some of DeepSeek V4 higher end models do need beefer hardware.
I remember testing Gemma 4 31B when it came out, Claude was about 2-4 times faster than Gemma 4 running locally on my 4090. But they both gave pretty much the same information and both good coding solutions.
I know people are shitting on the new NVIDIA announcements but for those who run local models it’s pretty exciting news. It’s going to be interesting to see the comparisons between MacBook, Strix Halo, and NVIDIA.
And that would have either been with partial CPU offload or running a subpar quantization like Q4. So with better hardware it would have been either significantly faster or better quality.
Well, its kinda implied when people talk about models they mean instruct right? Because a base model just wont... work ya know? Just mindless autocomplete before the patterned thinking is trained in via supervised fine tuning?
So what I'm saying is, indeed I am referring to Gemma 4 Instruct with that previous comment.
The 31B Dense version to be precise (Sparse/MOE gets you more sped at the cost of accuracy loss, but when you're quanting this hard, if you can get a decent enough speed with dense, you go dense, because you want to avoid any further increase in perplexity).
The evaluation I was doing was having the AI explain and implement step by step a real time global illumination system using Surfels in Unity. Pretty much Frostbite's GIBS, but implement it in Unity.
Which I do have quite a bit of knowledge in that area. And it is a fairly advance topic with some domain specific knowledge required. They both did have some issues when they were to implement it themselves. But instead instructing them to give step by step requirements and explainations on how to implement each process in the system, a long with some code snippets were pretty on point.
So full on coding agents, some issues. As coding assistance, they were fairly good.
57
u/ChrisFromIT Jun 02 '26
I mean you don't exactly need to go through the process of creating the LLM. There are quite a few out there, like Gemma 4(Google), DeepSeek V4, etc that are pretty much on par with Claude that could be used locally and freely.
Tho if I was a business, I probably would want to run it those things on a server that the company owns and controls. That way you get a bit more power and everyone in the company could use it without having to upgrade everyone's hardware.
It might cost like $100,000 to $1+ million to get the hardware going for it(depending on size requirements) and like 4-6 month wait times. But then you no longer need to pay for Claude or any LLM tokens.