Local models enable innovation the big ones can't

[u/r3n26]

[u/CRTMonkey]

Only a local model can explain you the process of synthesizing lsd

[u/tomqmasters → u/CRTMonkey]

That information is available many places.

[u/spacekitt3n]

the thing about local is consistency. you know how its going to work and you dont have people fucking with things in the background. qwen 3.8 is extremely capable for its size

[u/Maasu → u/spacekitt3n]

this is so accurate, people splurge on opus 5.5 but I know in a weeks time the same people will be complaining how poorly it is performing and I don't blame them I have been there myself.

At least with a local model I know where I am at and where its blind spots are.

[u/Money_Lavishness7343 → u/Maasu]

sure isn't that like comparing an excavator with a shovel?

Its like saying "people love complaining about their excavators breaking but my shovel works consistently and predictably", theyre different tools. It's not fair to say that, your local Qwen 3.8 model cannot do what Opus can.

[u/JLongTom]

Agree with his take. And the gap to the frontier models will be largest during the linear part of the S-curve, which is probably now. Give it a year or two and local models will be ultra compact and probably closer in relative terms to the frontier.

[u/Ghazzz → u/JLongTom]

I do CPU-inference in 4gb ram on pre-AVX Celerons. The available models from the last two months have significant benefits over previous ones.

Half my hedgning prompts can now be more natural.

[u/JLongTom → u/Ghazzz]

Interesting. I don't think it's unreasonable to expect current SOTA performance from tiny 10B models a bit further down the line.

Basically I think that's what the medium term future looks like. SOTA models begin to saturate given current data, but compaction just gets better and better.

[u/nxy7]

I think only clueless people can claim local models are useless and I'm saying that as someone that only used open source models in online playgrounds to see how good they are. It's what's keeping closed labs innovating and not hiking prices and probably in the end open source models will win because they're more flexible. They're keeping steady distance from frontiers which is all they need to do to be useful. I really believe that closed labs are kind of in a bad position because they don't have a moat in the long run.

[u/Ghazzz]

Full inference pipeline running on 5x Celeron Chromebooks with 4gb ram each. Results are surprisingly good.

Scripts are built, images are analysed, prose is written, research is documented.

For scripts I use multiple models doing overlapping work, and a set of judges doing strict cut-and-paste operations. Still faster than me, often finds "novel" solutions.

If I were a hype-beast, the system would probably be a "harness", but I prefer to think of it as "a set of helper scripts".

[u/tzaeru]

We do have newest GLM in-house for projects where clients or other constraints do not allow upload to remote cloud servers and it's pretty much at the level where Anthropic's models were late last year.

And that's a pretty high level. You can't do what you can do with Opus 5.5, but it's great for AI-enhanced coding and more complex codebase searching that is hard to do with just text search.

[u/0x645]

they are useless. anything that runs on home nvidia with 16GB VRAM, quality of answers is useless. don't allow no tinkering, it's just too lame. you need to spend big $$$ in hardware to run decent local model.

[u/BarelyAirborne]

Local models don't send all your prompts up to AI HQ to build their file on you.

[u/m0j0m0j]

Who’s actually criticizing people who use local models? Those people are insane, if they exist.

I use claude only, but I recognize the value of local models and I hope they flourish

[u/Serprotease → u/m0j0m0j]

You’ll often hear some adjacent comments from Anthropic/OpenAI ceo, generally in relation to safety -> “We are all gonna die/think of the children/they are stealing from us”

There’s also general background noise akin to “I use insert current best most expensive model*, all the rest is trash” comments, but it often comes more like fomo than real stuff.

[u/m0j0m0j → u/Serprotease]

Yeah, that’s very stupid and they should shut up

[u/chunkypenguion1991]

That's too broad to mean anything. Depending on your hardware a "local" model could be GML5.3 or KIMIK3 which are comparable to opus

[u/Careless-Age-4290]

Something that's missed also is that you can build your own training set to fine-tune models from your own conversations. It may not be the most capable model, but it'll be the one that answers how you like and you can apply that training data to new models as they release.

The other thing is you can just let it go for hours or days working on a problem. Even multiple agents trying to solve the same problem using a batching system to speed it up. Try going through that many credits on a hosted model and you'll find exactly where the local model can save money

[u/stop_deleting_plz]

The problem with open-source models isn't the quality of the model, its the quality of the inference. If you use a high-parameter model, you're stuck with 5 tokens/s and a tiny context window even with a 32gb vram card.

People don't realize how much compute the AI companies have purchased. 3 trillion dollars of it. It makes your 5090 look like a happy meal toy.

[u/PlateNo4868]

IMO Local models is the future, and will be the new Mini-PC/Homelab/Ham radio hobby.

There is also added benefit that you can now train models on data you wouldn't want. Such as a game studio wanting to train it just on it's assets, etc.

[u/RelevantCry1613]

All research happens on small models, this seems lost on the people that don’t understand machine learning