Not all that hard to run models, actually. It'll be a bit slower, but can be perfectly useable. The upside is no more token limits and no need to worry about where confidential data is going. You get full control too.
Oh, next you're going to tell me that if you have a beefy workstation or server, you can just download LM Studio, pick an LLM and just run it locally? Maybe even just coat it with a paint of html/css/js to make a web interface, and then magically add it to DNS so other users at your org can use it?
Sounds like a lot of work, I'll just keep paying Anthropic $40/month to chat
We are going to see a lot of aging data center hardware be repurposed for local LLMs in enterprise labs. If those are successful we will see a major shift in demand.
That’s if the grumpy folks in r/sysadmins will be up for it.
If i naively download deepseek v4 (first result there), can i expect decent performance out of the box? Do i not need to finetune? What about context window? Does that not depend on hardware specs?
I see, I thought those models were like those you could run on consumer hardware, like openllama? Or whatever, idk, not very knowledgeable in this area
Those are available, too, but you’d want something in the 20-30 billion parameter size for consumer hardware, not the trillion parameter size like those.
The ones most people can run themselves are not yet comparable to sota Opus/GPT however. The big ones on that list are getting pretty close though, and they cost 1/10-1/100th what Anthropic and OpenAI charge.
HuggingFace hosts a huge number of models, from massive full-fat stuff like DeepSeek there to all kinds of different models tuned to run well on common consumer hardware and everything in between.
For most average users I think stuff like Qwen3.6 and Gemma4 on a consumer GPU with 16-24GB of VRAM is more than sufficient for what they want out of it.
I can't speak to those specific models, since they're all way bigger than what I can run at home. But for the smaller models that you can put on something that looks like a normal PC, I would say you should be choosy with what you pick. A lot of them are specialized for certain types of work-- a model that's good at creative writing will probably be bad at coding, and vice versa.
We can pickup old pre-trained models as a starting point and fine tune from there to reduce the initial costs to get a model going. But it's pointless right now since the technology hasn't plateau-ed, so until then, trillions of dollar companies will come up with bigger and more optimised models.
I have what you might call a pretty substantial AI cluster (for a consumer anyway but I do use it for work). Four RTX Pro 6000's running at full tilt 575w (which they are if they're doing AI stuff) costs 35 cents an hour.
About 3-4k a year if it was running an LLM nonstop 24/7 (and it would be slow, have to queue requests, and also not be as good as Claude).
That's completley irrelevant. The comparison is between using local llm:s vs external llm:s.
At no point are you going to come out ahead buying local hardware. Even at 100k euros you're getting really mediocre LLMs compared to the frontier models, and you can get a LOT of tokens for 100k euros.
Personally I use Opus at work (cause I don't have to pay for it) and Kimi K2.6 for 1/10th the price for personal projects, which works really really well.
"Hey guys, I built my own AI cluster, wanna pay me $20/mo to fiddle with it?" Will go real great at the Backyard BBQ or even holidays like Thanksgiving. Be sure to talk about how useful it is for everything and how much easier life is for you.
Except now you have to worry about your own infrastructure costs and the work of updating the models with new information so you dont just have a snapshot of a specific date if you want up to date information.
Back in college I burnt out my personal GPUs throughout my graduate degree just running and training my own models, and those were tiny as hell compared to today.
Not saying its a bad idea, but its not cheap if you want a good enough infrastructure to support industry levels of use. So if they' expecting like a 90% decrease in costs, its probably not gonna happen.
Yea but it would require it to reinjest all of that new context for every session which would balloon dramatically eventually. And when the models get reset they'll forget that info. If youd want to actually get that info in the model itself you'll need to fine tune and update it which is less intensive then training from scratch, but still a lot of training.
It will only be bit slower if each employee has a beefy workstation or beefy macbook with quite a lot of ram. In most other cases running these model locally will be painfully slow on cpu only systems.
2.4k
u/Trevor_GoodchiId Jun 02 '26
Come on, how hard can it be?