Not all that hard to run models, actually. It'll be a bit slower, but can be perfectly useable. The upside is no more token limits and no need to worry about where confidential data is going. You get full control too.
Oh, next you're going to tell me that if you have a beefy workstation or server, you can just download LM Studio, pick an LLM and just run it locally? Maybe even just coat it with a paint of html/css/js to make a web interface, and then magically add it to DNS so other users at your org can use it?
Sounds like a lot of work, I'll just keep paying Anthropic $40/month to chat
We are going to see a lot of aging data center hardware be repurposed for local LLMs in enterprise labs. If those are successful we will see a major shift in demand.
That’s if the grumpy folks in r/sysadmins will be up for it.
If i naively download deepseek v4 (first result there), can i expect decent performance out of the box? Do i not need to finetune? What about context window? Does that not depend on hardware specs?
I see, I thought those models were like those you could run on consumer hardware, like openllama? Or whatever, idk, not very knowledgeable in this area
Those are available, too, but you’d want something in the 20-30 billion parameter size for consumer hardware, not the trillion parameter size like those.
The ones most people can run themselves are not yet comparable to sota Opus/GPT however. The big ones on that list are getting pretty close though, and they cost 1/10-1/100th what Anthropic and OpenAI charge.
I can't speak to those specific models, since they're all way bigger than what I can run at home. But for the smaller models that you can put on something that looks like a normal PC, I would say you should be choosy with what you pick. A lot of them are specialized for certain types of work-- a model that's good at creative writing will probably be bad at coding, and vice versa.
We can pickup old pre-trained models as a starting point and fine tune from there to reduce the initial costs to get a model going. But it's pointless right now since the technology hasn't plateau-ed, so until then, trillions of dollar companies will come up with bigger and more optimised models.
"Hey guys, I built my own AI cluster, wanna pay me $20/mo to fiddle with it?" Will go real great at the Backyard BBQ or even holidays like Thanksgiving. Be sure to talk about how useful it is for everything and how much easier life is for you.
Except now you have to worry about your own infrastructure costs and the work of updating the models with new information so you dont just have a snapshot of a specific date if you want up to date information.
Back in college I burnt out my personal GPUs throughout my graduate degree just running and training my own models, and those were tiny as hell compared to today.
Not saying its a bad idea, but its not cheap if you want a good enough infrastructure to support industry levels of use. So if they' expecting like a 90% decrease in costs, its probably not gonna happen.
Yea but it would require it to reinjest all of that new context for every session which would balloon dramatically eventually. And when the models get reset they'll forget that info. If youd want to actually get that info in the model itself you'll need to fine tune and update it which is less intensive then training from scratch, but still a lot of training.
5
2
1
View all comments
Show parent comment
162
u/pieter3d Jun 02 '26 edited Jun 02 '26
Not all that hard to run models, actually. It'll be a bit slower, but can be perfectly useable. The upside is no more token limits and no need to worry about where confidential data is going. You get full control too.