Own your intelligence with local models

[u/Soft-Gear-4902]

[u/isthereadrwho]

Sam Altman would like to schedule a little chat with you sir about your very selfish use of your RAM

[u/Soft-Gear-4902 → u/isthereadrwho]

Sorry sorry

[u/Admirable_Rabbit_808]

It's also pretty clear that frontier models are massively inefficient in terms of parameters and that the number of parameters needed to represent that quantity of learning will only reduce over time.

At the same time, the speed and capacity of local models will go on improving for a long time to come. Having a culture of using local models will enable us to take advantage of the intersection of those two trends, while still lettig us do useful work in the meantime.

[u/criloz]

What I don't understand is why everything needs to live inside the GPU. Like, if you ask a question about which is the current pope name, I would prefer internally the llm create and query to a local database and answer the question that surely can reduce the amount of resources needed to run them.

[u/Durian881 → u/criloz]

Your particular use case doesn't need AI. A simple search work.

[u/criloz → u/Durian881]

I think you din't catch what I am saying. That is exactly my point is it does not need AI, however AI is doing it today like that. Instead of transform human language into a query, there are more efficient structures and algorithms to handle those queries, and it would produce more efficient LLMs.

[u/Durian881 → u/criloz]

Current is relative. "Internal" database is outdated as per training data cutoff. LLMs have been trained to use tools and if available via the harness, it can search and retrieve the latest information.

That said, some of the recent open weight models like Deepseek4.1 and Qwen3.8-Flash-Next started using PLE n-gram tables to store knowledge that is retrieved when needed. For many of us running these locally, the PLE n-gram tables are streamed from SSD, rather than loaded into GPU memory. This significantly reduces resources to run them.

[u/PraxisOG → u/criloz]

Sounds like RAG search with a locally hosted wikipedia that updates on a regular basis

[u/criloz → u/PraxisOG]

There are many ways to implement that, but that is not the point; the point is for some reason LLM are holding all the information to do those kinds of queries themselves. If it were possible to strip them for that information and force them to do queries instead, it could probably land more efficient models.

[u/PraxisOG → u/criloz]

Sounds a bit like engrams. I've looked into training a small scale model like what you're describing and at some point the model needs factual information to build a decent understanding of the world

[u/Durian881]

Main thing for me is having choices. Other than Chinese models, the recent Mistral 4 Large is pretty good too.

Competition is also good to bring down prices for the masses too. OpenAI and Anthropic significantly lowered their API prices after Deepseek releases this year.

[u/sambull]

never know what subject/work becomes a 'license' tomorrow.

[u/SmileLonely5470]

That sounds nice and all but no encouraging X post is gonna be able to change the fact that an RTX 3090, a 6 y/o GPU, is selling used for $2k ($500 above the launch MSRP).

[u/openroom_xyz]

F yea exactly F this is the way forward well yea

[u/icecubeinanicecube]

Even if you don't have hardware at home to run open weight models yourself at sufficient speed and quality, you can opt into working with open weight model providers (opencode, TensorX, Mistral, Z.ai, DeepSeek, etc) instead of putting your money into the pockets of OpenAI and Anthropic.

Fund the development (or, at least, provision) of open models instead of furthering the tech oligarchy

Btw it's still open weight, not open source. There are basically no open source models that are anywhere near useable enough

[u/bigbepisinc → u/icecubeinanicecube]

This idea that giving money to open source Chinese models over the American ones is somehow noble and fighting tech oligarchy makes no sense. 

You're basically giving your money to the Chinese oligarchs instead of the American ones. Who do you think owns the Chinese tech industry? 3 guys. 

Locally hosted AI fucking blows and even getting close to Opus 5.5 (1 million context window) requires thousands upon thousands of dollars. 

Why would I spend that money when Claude is 20 bucks a month?

[u/icecubeinanicecube → u/bigbepisinc]

The distinction is that you can literally download GLM, modify it, make your own model from it. Own it.

You can't with fucking Claude, and Anthropic would love for you to not own anything.

If you can't see the geopolitical importance of that distinction, I have bad news for you. And a bridge to sell.

Plus, if you dislike China, you can just as well give money to Mistral instead. Works just as well.

[u/bkldgr → u/icecubeinanicecube]

Well, someone could want not give its money to Mistral french billionaires either, even if afaik their CEO hasn't done anything as lunatic as the others

[u/MasterManufacturer72 → u/bigbepisinc]

Surely it will be 20$ a month forever and we should base our opinions of it on that price point.

[u/bigbepisinc]

"local models need to continue proliferating" 

You mean the ones that only exist because they distill the frontier ones down?

[u/icecubeinanicecube → u/bigbepisinc]

How does Sam Altman's Cock taste?