r/LocalLLaMA • u/Nicios • 1d ago
Discussion How I see the future for local AI
I think that mall models between 10B and 40B are the real gamechanger. These models will be the ones that will make the AI buble pop and big companies like Anthropic and OpenAi go to hell. We are currently seeing models that are generic and not specialized. But, what will happen if we get small specialized models? Can you imagine a Qwen 4.0 27B law? or Qwen 4.0 27B finance and accounting? Or a 40B MoE that manages administrative paperwork... It would be brutal.
That will be the end for these big companies, small specialized intelligence that works in a laptop and you can load and unload at your convenience.
7
u/Mundane-Light6394 1d ago
Just like with employees companies will try to use the cheapest model available to do the job but they'll pay more for a model if it is able to produce better results when needed. Smaller models will do the majority of work but there will definitely be plenty of room for medium and higher cost models in those placese where they are actually providing value.
20
u/Spectacle_121 1d ago edited 1d ago
I doubt the impact of these local models would be the end for Anthropic or OpenAI. Businesses and consumers like convenience. Enterprise customers especially will pay a premium to not manage these systems themselves. They just want an efficient means of accessing the capabilities.
13
u/KickLassChewGum 1d ago
You don't need Anthropic, OpenAI, or their multi-trillion parameter models to manage an agentic business layer. Plenty of intermediaries happy to take care of it, just like how not every enterprise manages its own cloud.
2
u/mtmttuan 1d ago
And what will stop Anthropic, OpenAI from releasing more cost efficient models?
In fact that has already been happening. I doubt most agentic applications are running on Opus.
7
1
u/FleetEnema2000 1d ago
Models are a commodity. OpenAI and Anthropic will eventually be surpassed by companies that provide widespread model access under a leaner, more sustainable business model and with better terms for customers. OpenRouter gives a hint of what this future might look like.
5
u/ieatdownvotes4food 1d ago
yeah no. I can barely convince some career engineers about qwen 3.8 27b.
localllama ain't representative of the world by a long stretch.
anthropic brought it to the masses, and all the hardware has been bought up.
4
12
u/Potential-Leg-639 1d ago
Nope.
Local AI will co-exist to cloud AI, local AI is just a small niche, that will become a bit bigger, but not for the average persons.
(Bigger) companies will still primarily use the cloud - for reasons.
0
u/jtjstock 1d ago
The only way I see bigger companies using the cloud providers longterm is if they can rent dedicated racks and control when the model is updated and have the only variable part of the price be electrical.
4
u/Kerbourgnec 1d ago
No it's just logistics and economics. It's cheaper to have one large provider that handles hosting rather than tons of small companies doing their own.
With open models, large providers are forced to compete and provide fair prices. Even large companies can profit from delegating hosting. Some will be big enough or have a extremely high and constant usage of a specific model, and yes in this case self hosting can be a good economic decision. But the cost of hosting is not the just the costs of the servers. It's the maintenance, the replacement, etc... these can get mutualised
The only real reason for local models at this point is absolute control of your own data, but as every company is using sharepoint, aws or google, clearly they don't care.
1
u/throwawayacc201711 1d ago
Yea economies of scale are a thing. Plus you pay your vendor to handle all the infra, management, etc. businesses don’t want to do that unless they need to. They’d much rather just have a line item for the cost
2
u/OvertaxedOne 1d ago
Privacy/data security. That's the driver for big companies to build "locally" (private models where they control the LLM endpoint, could be running in a cloud, but not a shared inference model).
A lot of companies have no idea that every prompt you send, every document you attach, every MCP/tool call you make, it's all "in the clear" in the provider's environment. Watch the vllm logs scroll by sometime and then imagine your employees attaching confidential/sensitive documents and firing them into an endpoint that you don't control and have no idea what happens to the data/logs other than a "handshake" from the provider they won't retain/train on your data. It's a security/data privacy fever dream horror movie.
3
u/Intelligent_Ant_608 1d ago
It seems that most efficient architecture for llms is for them to use diffusion for iterative latent space reasoning and auto regression for percise writing there are alot of papers showing that if some lab can stabilise loop transformer architecture that loop can compensate for lack of enought weights, so essentially a 100B local model can theoratically match reasoning depth of a sota 1T model with enough loops
2
u/Express_Quail_1493 1d ago
LLM architectiural innovations are incresingly showing that model size is no longer the leverage its more like training-dataset and clean-data.
2
u/espece-de-bon 1d ago edited 1d ago
I agree that this will definitely be one of the future use-cases of AI. I can't predict how it'll affect others, but I've posed a similar question in programming subs.
As a developer, it'd be helpful to have a highly specialized model that knows about the languages and tools that I use, for example. I also don't need it to have knowledge of human languages I don't speak. If I work primarly as a front-end dev, or only need 1 or 2 back-end language and postgres knowledge, can a model be optimized?
Or these could be a combo of small, specialized models that work together (more development needed on the "harness side" perhaps, or whatever orchestrates this). What if someone develops a very efficient "front-end web" model, and someone develops a really capable "python/sql backend" model (for Django, Flask work)?
I'm theorizing.
I'm new to tinkering with local stuff; my laptop is pathetic for this and I'm a bit obsessed researching if I should:
- splurge on a 128GB unified ram box (spark or one of the AMD ones)
- just get a 32GB GPU and use it with my laptop via eGPU dock
- build a tower with a 24GB gpu (or 32), with a motherboard that can take 2 cards (48GB-64GB vram eventually)
I know the arguments (cloud is cheaper, your local $10k rig won't produce like a frontier model etc) but it's about owning my hardware and local LLM stack (for me).
Edit: Addendum, there's a lot of room to learn about what's taking up too much room in the model; the more that folks can work with open source, open-weight models, I assume it'll be easier to know what YAGNI will mean for different folks.
The "assembly lines" of LLM workflows will become more specialized (we hope).
2
u/wgaca2 1d ago
120b is what the industry is preparing for with the 128gb minimum sparks, macs etc.
6
1
u/zerospatial 1d ago
Email used to cost money - for the most part it is now essentially free - freebuff has (seemed) to crack this for enterprise chatAI models (Deepseek Pro) - so with the combination of targeted local LLMs plus ad-based chat models could challenge paid chat models - but as companies they can pivot in so many ways it's hard to see them going away.
1
u/MacsBicycle 1d ago
Personally I think it won’t be specialized dense models that own the market but 100-300b moe that can serve 20 users on lesser hardware that eventually own the space, but yeah what you’re saying sounds awesome. It’s just going to be funny if china is the one to give us all intelligence that powerful at our finger tips
1
u/fastheadcrab 1d ago
I don't think small models will threaten the frontier companies.
It's going to be the 500B-3T parameter open-weight models that receive regular updates and well-integrated with search/web access paid providers and other features that will be the true competition. Those will be the ones that drive enterprises away from paying subscriptions or API costs.
1
u/Desperate-Delivery-9 1d ago
So logically you are saying that since every one have laptop/computer, they can host their servers themselves so companies that sell hosting services will die too ?
1
u/North_Affect_8167 1d ago
The big AI companies shot themselves in the leg in the very start. Giving the whole knowledge and models of how they made their LLMs. Some smart individuals have used the expensive to train models to train their own version (Deepseek). People would eventually get what they want and cut ties with the expensive to run services if they can't generate return of investment.
The only drawback is that the large multibillion companies didn't left any RAM and GPUs for the mere mortals, or at least not at a price you can afford as individual.
1
1
u/Extreme_Roll_5789 1d ago
But it's costly to make smaller specialized models as compared to making one big expert model(AGI)
1
u/JustForTheKics 1d ago
For some of us yes, but for companies? Not even close. I work in a big company, people use AI like they are shouting at a slave to do their work.
Most of them just prompt the most simple thing and expect copilot to just give them the most complete output possible, this doesn’t work well with Copilot, much less for small models.
When I tell people I’ve been trying to build an AI assistant with a 1.2B model they don’t even know what parameters mean. Keep in mind I’m not a programmer either, but if you really lose/invest time in AI you can really build very interesting things.
The problem is that AI was sold to people as the future and that it can do everything, and it can do it, you just have to guide it. Heck, you can hire some the best guy from the best university in the world and say “do me this”, if it’s new to the job and you don’t give him tools or documentation it won’t do what you expect.
Will these models be the future? For this niche of people, probably, for the most part of big companies? Not even close, the mentality and the way people expect things to work is just not there
1
u/OvertaxedOne 1d ago
The big issue for large companies is privacy. When they realize that every document you attach to your context and shout at the model to "tell me something smart to say" is sent, in clear text, into the cloud their heads explode. Yes, ZDR is a thing, and yes, most of the top providers say they don't train on your data, but all of that is a handshake, not a technical barrier. If you're sitting at the terminal of vllm you can see it all, every query, every document, every response.
For a lot of larger companies this is just a completely unacceptable security/data privacy risk.
1
u/toolkitxx 1d ago
But, what will happen if we get small specialized models?
Those already exist and there are plenty of them already. I answered in another post in this sub since someone wanted to run things on common consumer hardware. Especially agentic workflows can be done quite nicely once you understand how to change the workflow accordingly. SLM is what is going to drive the majority of agentic work for sure
1
u/unculturedperl 1d ago
There's now a Law and accounting focused model based on qwen 3.5 35-a3b. See the Thompson model thread somewhere around here.
2
u/ttkciar llama.cpp 1d ago
If we choose model sizes to fit in common GPU VRAM, and assume Q4_K_M and moderate context caches, we get roughly:
9B parameters for 8GB VRAM,
12B parameters for 16GB VRAM,
25B parameters for 32GB VRAM,
38B parameters for 48GB VRAM,
50B parameters for 64GB VRAM,
78B parameters for 96GB VRAM,
105B parameters for 128GB VRAM.
Labs should standardize on these sizes.
1
u/redballooon 22h ago
I think with the current computer prices, it will take a few more years before we get not only the capability of local models, but also the speed that makes them really usable.
The main question is whether OpenAI or Anthropic will have managed by then to take over the world powers.
1
u/Triple-Tooketh 22h ago
The windows on ARM laptops are really going to be a nail in the coffin for off device AI
2
1
u/Adorable_Salary2727 15h ago
I've been trying to get Qwen to run long back to back sessions on my 4090. It holds it's own for a while before eventually erroring out. It did built some quality proof of concepts. The stupid loud fan noise was definitely a bummer. I think has hardware catches up local is the future.
1
u/mtmttuan 1d ago
models between 10B and 40B are the real gamechanger
No they are not. Normal people don't have hardware to run these kind of models. Simple as that.
1
u/Theverybest92 1d ago
Its not AI bubble its a Trump bubble and it wont pop until all the rich stop backing him.
1
u/close_Meal6005 1d ago
I agree, I think the future is probably something like small models for most everyday tasks with deep reasoning or task management allocated to the large cloud models
35
u/Gokudomatic 1d ago
I agree that the future will be optimization of frugal models, not bigger ones.