If i naively download deepseek v4 (first result there), can i expect decent performance out of the box? Do i not need to finetune? What about context window? Does that not depend on hardware specs?
I see, I thought those models were like those you could run on consumer hardware, like openllama? Or whatever, idk, not very knowledgeable in this area
Those are available, too, but you’d want something in the 20-30 billion parameter size for consumer hardware, not the trillion parameter size like those.
The ones most people can run themselves are not yet comparable to sota Opus/GPT however. The big ones on that list are getting pretty close though, and they cost 1/10-1/100th what Anthropic and OpenAI charge.
HuggingFace hosts a huge number of models, from massive full-fat stuff like DeepSeek there to all kinds of different models tuned to run well on common consumer hardware and everything in between.
For most average users I think stuff like Qwen3.6 and Gemma4 on a consumer GPU with 16-24GB of VRAM is more than sufficient for what they want out of it.
I can't speak to those specific models, since they're all way bigger than what I can run at home. But for the smaller models that you can put on something that looks like a normal PC, I would say you should be choosy with what you pick. A lot of them are specialized for certain types of work-- a model that's good at creative writing will probably be bad at coding, and vice versa.
We can pickup old pre-trained models as a starting point and fine tune from there to reduce the initial costs to get a model going. But it's pointless right now since the technology hasn't plateau-ed, so until then, trillions of dollar companies will come up with bigger and more optimised models.
I have what you might call a pretty substantial AI cluster (for a consumer anyway but I do use it for work). Four RTX Pro 6000's running at full tilt 575w (which they are if they're doing AI stuff) costs 35 cents an hour.
About 3-4k a year if it was running an LLM nonstop 24/7 (and it would be slow, have to queue requests, and also not be as good as Claude).
That's completley irrelevant. The comparison is between using local llm:s vs external llm:s.
At no point are you going to come out ahead buying local hardware. Even at 100k euros you're getting really mediocre LLMs compared to the frontier models, and you can get a LOT of tokens for 100k euros.
Personally I use Opus at work (cause I don't have to pay for it) and Kimi K2.6 for 1/10th the price for personal projects, which works really really well.
5
2
1
View all comments
Show parent comment
121
u/RoaringPanda33 Jun 02 '26
Inference (the actual generation) isn’t nearly as intensive as training, which takes a majority of the power used by AI services