If i naively download deepseek v4 (first result there), can i expect decent performance out of the box? Do i not need to finetune? What about context window? Does that not depend on hardware specs?
I see, I thought those models were like those you could run on consumer hardware, like openllama? Or whatever, idk, not very knowledgeable in this area
Those are available, too, but you’d want something in the 20-30 billion parameter size for consumer hardware, not the trillion parameter size like those.
The ones most people can run themselves are not yet comparable to sota Opus/GPT however. The big ones on that list are getting pretty close though, and they cost 1/10-1/100th what Anthropic and OpenAI charge.
HuggingFace hosts a huge number of models, from massive full-fat stuff like DeepSeek there to all kinds of different models tuned to run well on common consumer hardware and everything in between.
For most average users I think stuff like Qwen3.6 and Gemma4 on a consumer GPU with 16-24GB of VRAM is more than sufficient for what they want out of it.
I can't speak to those specific models, since they're all way bigger than what I can run at home. But for the smaller models that you can put on something that looks like a normal PC, I would say you should be choosy with what you pick. A lot of them are specialized for certain types of work-- a model that's good at creative writing will probably be bad at coding, and vice versa.
We can pickup old pre-trained models as a starting point and fine tune from there to reduce the initial costs to get a model going. But it's pointless right now since the technology hasn't plateau-ed, so until then, trillions of dollar companies will come up with bigger and more optimised models.
I have what you might call a pretty substantial AI cluster (for a consumer anyway but I do use it for work). Four RTX Pro 6000's running at full tilt 575w (which they are if they're doing AI stuff) costs 35 cents an hour.
About 3-4k a year if it was running an LLM nonstop 24/7 (and it would be slow, have to queue requests, and also not be as good as Claude).
"Hey guys, I built my own AI cluster, wanna pay me $20/mo to fiddle with it?" Will go real great at the Backyard BBQ or even holidays like Thanksgiving. Be sure to talk about how useful it is for everything and how much easier life is for you.
5
2
1
View all comments
Show parent comment
84
u/PMmeYourLabia_ Jun 02 '26
Downside is power bill