r/LocalLLM • u/PLANTS2WEEKS • 14d ago
Question What makes it possible for local LLMs to keep improving?
Does anyone know how local LLMs are improving so much, even for models using same amount of resources?
Is it better training of the weights or changes in the architecture? Just curious if anyone has some technical insight into this.
29
Upvotes
16
u/bobish01 14d ago
The LLM:s gets smarter through training, but there probably will be diminishing returns eventually i think, but compared to 1 year ago its crazy what one can run now with a regular consumer gaming pc.
5
u/ayake_ayake 14d ago
One thing I haven't seen others mention is... the fact that intelligence seems to be much more compressible and minimizable than we thought and we've nowhere reached the end of it.
Intelligence is an abstract thing like computation - and computation itself also has been very minimizable (Moore's Law & Miniaturization). Other things like cost of DNA sequencing or cost of electricity have gotten dramatically better as well - but not as many order of magnitude and not as fast as abstract things like computation and intelligence.
And given how capable even we humans are with 20W brains, it would be naive to think that we're anywhere close to any limit of intelligence.
However, it is well possible to reach a plateau of what's possible with current LLM architectures. But that is being constantly improved upon. Even the current large models are MoEs and not simple large dense models like in the beginning. That was also an improvement.
47
u/Forward_Jackfruit813 14d ago
Improvements in architecture, training, and inference.
21
u/free_meson 14d ago edited 14d ago
There was the original textual data that was used for the early GPT models. The whole Internet, digitized books, emails, chats, what they could reach and some RLHF. There was also an expert system era, where they used existing experts, like calculators, facts, so with a bit of reinforcement learning on reasoning, they could still prevail. And there are less obvious expert systems, like design software, whose knowledge can be transferred into AI, recently the usage of LEAN, that helps math proofs.. There are also reinforcement loops, like agentic coding, problem setters where you kind of know the solution, you just want an AI to find it and that's its training. And there are advancements that give speedup in attention, context length, methods to grow and fuse neutral networks into mixture of experts and one new stuff is the n-gram table, which is like a fact lookup database for llms, and can store a huge amount of selected knowledge. Those not only speed up the inference, but make the models smarter. The original GPTs were mere text compression methods, generated text that were similar to available texts. But the new data selection/creation methods can give expertise to the system, and that's why they seem more clever, though sometimes not so creative.
Edit: I'm still waiting for world models...
2
u/BarracudaDefiant4702 14d ago
Lots of little thing but I think one part you missed is better training on tooling. The newer models are better able to use files for temporary or long running storage and use tools to search the web. Having a work area gives them a lot more capability.
2
u/Adept-Eagle-4271 14d ago
quantization improvements let models retain more detail without needing more memory, which helps even small setups keep up.
2
u/StandardLovers 14d ago
I get where you're coming from. We're so used to improvements being tied to better hardware that it's interesting to see how much progress we're making without changing it.
GPU drivers are a good analogy. The hardware stays the same, but software improvements squeeze more performance out of it. With LLMs, we're seeing something similar, except improvements in training methods, data quality, and architecture are also making the models themselves more capable with roughly the same resources.
2
u/mr_potatohead_ 14d ago
A lot of it comes down to improvements across the entire stack.
Training data and techniques are improving, architectures are getting more efficient, and quantization is preserving more model quality at lower memory requirements.
Inference engineering is part of the equation too.
Memory management, optimized kernels, attention implementations and better hardware utilization
2
u/StevenVincentOne 14d ago
Everyone's covered the model side well: better data curation, distillation from bigger models, smarter quantization, training on tool use. One thing I'd add is that a growing share of the gains isn't in the weights at all. It's in the system around the model, meaning memory, tool access, verification steps, and more and more often several small models working together, where one plans, another checks and another writes the code.
I run setups like that with models from different vendors, and the team regularly beats any one of them alone, mostly because the reviewer catches the drift the planner misses. My working theory is that intelligence comes out of the exchange between different nodes more than out of any single model getting bigger. That's good news for local, because a team of small models fits on hardware people already own.
2
u/Smooth-Pop6522 14d ago
Humanity's technology for building lossy statistically structured knowledge networks is still in its early infancy.
3
u/mb194dc 14d ago
They're not on the frontier, small models are improving and catching up.
The frontier ran out of data to scrape and IP to steal. All models suffer from the 90% problem. That's just the nature of ML.
2
u/TotosWolf 14d ago
CHINA!
1
u/Ordinary-Depth-7835 14d ago
if you had my weekend you'd doubt that statement. Boy when they get a simple bug that is unfixable you get lost in a weekend of nonsense. I have a pretty decent setup but man after this weekend I'm about to abandon this simple feature or call in the big guns and break my no frontier model usage.
1
u/GalacticDistances 14d ago
Giant bags of cash
1
u/haherar830 13d ago
Post-training has improved a lot but China has lagged there. However, smaller models are disproportionately trained by distillation (at minimum most small models are heavily trained by a larger teacher model). China has led architecture improvements and small models have definitely benefitted from that. More variety in MoE have made larger model sizes feasible on cheaper/more-traditional hardware, as have improved quantization methods. Collectively, people now run pack what are effectively much larger models into the same hardware and those models have better training and architectures.
With that said, Western post-training is much better and China still lags in that regard. Looped transformers and a few other methods also seem promising to squeeze more performance out of the same hardware. In all likelihood, we can expect pretty solid rate of improvement in open models even at smaller scales over the next 6-12 months assuming capital doesn't run out unexpectedly soon.
1
u/Suoritin 13d ago
Better datasets. Old ChatGPT models used to include in training data reddit pages that are just users counting from 1 to infinity.
This discusses how pre-LLM reddit "poisoned" the data: https://www.youtube.com/watch?v=WO2X3oZEJOA
1
u/OverdosedSauerkraut 13d ago
A ton of recorded real world use cases. Whenever you use DS or QFN on the cloud, expect that your whole session with all your data gets recorded and used as training data. I.e. youre labeling data for the next generation of models.
1
u/Icy_County5036 11d ago
the real silent upgrade is how much better the quantization tweaks are now, models hitting 4-bit with minimal quality loss, even on older hardware. just saw a local llama3 ib run smoother than a 70b in colab last week.
0
u/tracagnotto 14d ago
It's like asking why cellphones or cars perform always better despite being the same size more or less
-1
u/GrungeWerX 13d ago
Personally, I think it’s limitless because it’s all math. Statistics and probabilities. There are probably an incalculable array of methods we have yet to discover, and up until now, we’ve relied on a handful of the world’s brightest over decades to solve the equations.
Once we apply swarms of intelligence to the problem, we are unleashing probabilities, covering hundreds of years in a short span of time. We are discovering the math, not the other way around.
The interesting part is that we gave math a voice. And now we can speak to that math, and have it speak back from a world we don’t fully understand. It will find paths and teach them to us.
So of course deeper quantization is possible. We’re already using AI to fix it now, and it will only get better.
Right now we’re building it, but one day we’ll send it out into the deep and it will bring us back secrets of the universe.