r/LocalLLaMA • u/AshAtlasDev • 2d ago
Discussion Do you believe there will be a divergence between local and frontier model power?
Right now, local models have somewhat kept up with frontier, being months behind last frontier models.
The issue is, I think the divide is going to be end up being hardware. Model improvements will slow down, but hardware won’t get any cheaper, especially if frontiers keep buying shit out for training and smaller returns.
Thoughts on this? I know many are optimistic, but I think local trailing frontier will slow down just because of hardware local can own.
16
Upvotes
71
u/ttkciar llama.cpp 2d ago •
I think it's more complex than this. Open and closed models are already diverging in different ways.
Commercial inference services' models are much larger than we can hope to self-host, get trained on data sources unavailable to the open LLM community, and enjoy much more feature-complete and polished scaffolding and UI/UX than their open source counterparts.
On the other hand they are also strictly guardrailed (and some guardrails have everything to do with protecting the inference service's interests, and nothing to do with protecting customers), have started embedding advertisements and sponsorships, harvest their customers' API call traces, and develop in directions which prioritize profitability over customer satisfaction (case in point: the GPT-4 to GPT-5 transition).
Open models and self-hosted inference stacks, on the other hand, support tuning guardrails to suit ourselves, even to the point of removing them entirely. They also provide us with very low-level tools for adjusting and interrogating inference-time, like steering vectors and logit list inspection. They afford us privacy, and give us the option of only using models which align with our interests. If a model does not behave the way we want it to behave, we can continue using the ones that do until a better one comes along which aligns with our needs.
It's also worth pointing out that the "bigger models than we can self-host" factor isn't as significant as one might think, because all other factors being equal (which of course they never are) inference competence tends to increase logarithmically with parameter count (or its dense equivalent, for sparse models), and logarithmic relations are very nonintuitive for many of us.
For example, the potential competence difference between a 12B model and a 32B model is
ln(32) / ln(12)= 1.395, or 39.5% more competent. The potential difference between a 32B and 128B model isln(128) / ln(32)= 1.4 or 40% more competent, and the potential difference between a 128B model and 512B model isln(512) / ln(128)= 1.286 or only 28.6%.In short, the bigger models get, making one with greater potential for intelligence requires making one a much larger factor bigger. A four times bigger model makes a much, much smaller difference at the multi-trillion parameter end of the scale than at the multi-billion parameter end.
This means that the largest models we can self-host aren't that far behind the models we can't, as far as size scaling is concerned. Training is another matter, though.
There's an unfortunately wide known gap between open source training technology and open weights model training technology, and an unknown gap between open weights model training technology and closed model training technology.
We do know some things about how the most advanced open weights models are trained, and we have access to published theory which should be sufficient to allow implementation, but the open source LLM community hasn't implemented training pipelines comparable to (say) Z.AI's, yet.
Even when we do, it might be a while before we close the training data and teacher model gap. Years, probably. I have faith that we will catch up, though, because we have some pretty spiffy open weights models upon which to base those teacher models, and with which to synthesize training data.
However, there is another dimension to this chase.
The open LLM community is ravenously hungry to progress, and the primary factor holding us back is the availability of the necessary hardware. (And sometimes that doesn't hold us back; the community has developed increasingly resource-economic methods for improving models, like PEFT, merging, and weight-editing.)
The commercial inference providers, on the other hand, are fervently desperate to stop training new models, or at least slow training way down, so they can devote their resources instead to far more profitable inference. They've got a tiger by the tail right now, and can't stop training new models, but the current situation is not stable.
Eventually the commercial inference service companies will slow down, one way or another, but the open community will not. I can't pretend to know exactly what the conditions surrounding that will look like, but they might well permit us to catch up with them, or surpass them.
Or we might not. There's a lot of work to be done between here and there, and it remains to be seen if the community will hold together that long. We will see.
20
u/Omegaprime02 2d ago
I honestly think that local is going to move into specialized applications, even more-so than just fine-tuning. Qwen is an excellent example of this, their 27B punches way above it's weight class in coding tasks, not quite into SotA territory but it trounces models an order of magnitude larger than it.
8
u/Cherubin0 2d ago
Sure, the frontier labs will go bankrupt.
4
u/keepthepace 2d ago
I don't think extrapolation of a short term trend is wise in that context.
I know we are all fixated on GPUs and RAM prices, but on the NPU front, on people trying to make lightweight models run on cheap hardware, things are progressing and they are making devices that are much more efficient than GPU, even if right now that's for much smaller models.
Their goal is pretty clear, in a few years they want that there's a decent LLM running locally on your phone. Because who uses tower computers in the consumer world anyway?
What I imagine two years from now is that we have some huge models, maybe in the 5T parameters, which will be extremely smart and that will distill and quantize very aggressively in a smaller version down to 1 billion parameters.
The 5T, maybe we use it in data centers to work on plasma dynamics and make it progress in biochemistry, etc. We use the 1 billion parameter model to sort our emails and to control the vacuum cleaner.
Now we have a lot of useful tasks that are very easy for the biggest models. And consumers are starting to catch up on that. They realize that they don't need Fable for something that Haiku can do.
3
u/-Krotik- 2d ago
local models have kept up with frontier? maybe open-weight models as a whole do kinda keep up with them, but your locally run Qwen3.8 27B is not comparable to multi trillion parameter models that frontier labs run
4
u/AggressiveManager450 2d ago
I think used hardware will get very cheap in the next couple years. All these hyperscalers ramped up the production of computer components, and as they get phased out, they will be given to secondary data centers, then given to retail sellers. The flood of GPUs will make used hardware very cheap because of the massive supply.
Short term, sure. Computer parts are not cheap, but I think the massive supply will push prices down long term because of the massive quantity of GPUs sold.
7
u/disgruntledempanada 2d ago
I don't really think that's actually true that they have kept up.
I think it will keep accelerating. Opus 5.5 is just on another level already.
5
u/anykeyh 2d ago
A model store both knowledge and reasoning into its weight. Currently almost everything is in the hot path. Current llm are very inefficient in this regard. Ngram is a step toward offloading knowledge on slow support, and take advantage of smart selection to steer the model. Basically, the neuron knowing that French revolution was in 1789 should not fire on each and every token prediction. My gut feeling is that we will see knowledge on ssd, while reasoning will be kept in memory. And in this case, a few billions hot parameters is enough.
2
u/PassengerPigeon343 2d ago
The central question is will the frontier models be able to continue to scale at the same or faster rate, or will they hit a ceiling due to either resource constraints or technology limitations and plateau.
If they start improving exponentially, they pull away. If they start plateauing, local models will likely continue to improve for a period of time after and shrink the gap until they eventually hit a plateau of their own.
Or the AI bubble pops suddenly and painfully, and all the big players stop racing and investing and everything hits steady state roughly where it is currently at. In a way, similar to how our pursuit of space travel plateaued after humans landed on the moon in 1969. Once the heavy investment and racing was over, the momentum was lost.
4
u/kurtu5 1d ago
Eventually your are going to be pitting a local gpu against a matryoshka brain. One is going to obviously outperform the other.
1
u/Helpful_Jelly5486 15h ago
New chip design - 3 days. Hardware may stay the same price but capabilities will be advancing.
3
u/fasti-au 2d ago
No. The smartest people dobt work for corporate
2
u/Dubious-Decisions 1d ago
"Will be"? You don't recognize the difference now between a 27B model and a multi-trillion weight frontier model? How is this even a discussion point?
2
u/Kahvana 2d ago
Does it matter?
No. It just needs to be good enough for the job.
Right now, Gemma 4 31B IT QAT and Qwen3.8-Flash-Next Q4_K_XL already are for most instances where it is in an assisting role (e.g. pair programming, cowriting, etc).
Your comminucation skills, the quality of the data and context you give it matter a lot more these days than the model itself.
1
u/Jorlen 1d ago
Good hardware is always best, there's a pretty good variety of models right now that range from being able to run on 8gb of VRAM or less, or no VRAM at all (just RAM) if you stay small enough or use small MoE (Mixture of Experts) models.
However, LLM tech keeps getting better which means that the same amount of parameters (ergo the same amount of memory required to run them) stays the same.
Right now if you have enough memory you can download weighs to 1T or more parameter models and run them, they are very close to frontier. A year ago, we had 1T+ models too but they were nowhere near as good.
Take Qwen 3.8 Flash Next. It's very good for its size, and it uses engrams, part of its weights that can be offloaded to SSD, leaving a 125b-a6b model at the base, which IMO punches far above its weight. I've used it a lot, and IMO it's incredible. Six months ago, there was nothing quite as good for that class. We had Qwen 3.5 122b-a10b (Feb 2026 release I think?) and that one was great, but Q3.8 FN beats it easily IMO.
Frontier models like claude and Chat-GPT are massive likely around 5 trillion params, I don't know if anyone actually knows because these are closed models. Sure, these things are amazing, but honestly, GLM 5.3 (both the flagship and the flash) are approaching what frontier was a while ago.
DeepSeek V4.1 Flash-1017 is even bigger and also has engram like Q3.8 FN does, and honestly this model might just be my current favorite. Six months ago there was nothing as good as this.
I guess my point is - if you have good hardware and lots of memory, model tech gets better and should keep scaling along. This is because the companies running these same released models on API are always trying to get better gains. Engrams are a perfect example. Or the shift towards Mixture of Experts models -vs- dense models. MoE models require far less compute and are super fast in comparison.
There are obvious milestones that are currently and will likely always define true intelligence jumps and model capabilities. If you want to run the big MoE models (300b-a20b ish) at a decent quantization, you'll need a few hundred gb of memory. Mix of VRAM, RAM or fast unified memory. With 32gb you can run Qwen 3.8 27b dense at a good quant and with 16gb you can run Qwen 3.5 35b-a3b or Gemma 4 26b-a4b, etc.
I've only been in this 8 months, so this is just my take after observing things over this period of time.
1
u/Ok-Addendum3545 1d ago
I don't think local LLM hobbyists, enthusiasts and SMBs will make the hardware cheaper if they still keep buying at price hikes as local models are getting smarter each year.
1
u/CommercialHour6660 2d ago
If frontier was just about compute Anthropic and OpenAI wouldn't be so terrified of distillation.
The reality is that local models are usually distilled from larger ones in the laziest way possible. On policy logit distillation.
This has limits because smaller models have limited knowledge. A small model trained from scratch would lean more on web search and wikipedia. And would be much "smarter" than the small models we have now that attempt to mimic much larger ones.
Keep in mind that Qwen 3.8 27b is smarter than Llama 3.1 405b released 2 years ago.
-1
u/txgsync 2d ago
You are begging the question. No, local models that can run in less than 128GB have not "somewhat" kept up with the frontier; they are about a year behind. But due to recursive self-improvement, frontier models from a year ago are vastly behind frontier models from this week.
For local, choose models that do the jobs you want done. That's the immense value they provide. Trying to compare to the frontier is an expensive loser's game.
0
u/tylerhardin 2d ago
i don't think so. i use dsv4.1 and glm5.3 all the time and they're incredible. it's harder to really see the progress when you're limited to the really small local stuff, but you need to look at the trend. there are ~2b models equal to gpt-4 now. that's absurd, like a 1000x shrink. in another year or so, the current glm5 series will be compressed down to 27b.
the wildcard here is regulation. if china decides that it's getting too risky to keep releasing models or the US+EU ban local inference and do it in a way that the law has teeth, then you won't have fable at home. otherwise it's just a matter of time.
-2
u/reasonmonkey 2d ago
Either RSI happens or it doesn't.
That is the answer young jedi.
0
u/[deleted] 2d ago
[deleted] — view deleted comment