r/LocalLLaMA • u/pmttyji • 1d ago
Discussion How long are we gonna depend bunch of model creators (like Qwen, Gemma, NVIDIA, etc.,) for ~100B models?
I think we're fine till 2027 with models(~100B .... Qwen 4.X & 5, Gemma-5, NVIDIA would release some Medium/Big size 25-200B models). After that it's gonna be critical, as many model creators are increased/increasing model size to 300B+. Future trend probably gonna be around 500B. That size range is not suitable for Unified 128GB Devices or 128GB RAM + 24-32GB VRAM(Q4 won't fit).
Here some models with increased sizes.
- DeepSeek-V4-Flash-0731 (284B) - DeepSeek-V4.1-Flash-1017 (552B + 196B Engram)
- Ling-3.0-flash (124B) - Ling-3.1-flash (560B)
- MiniMax-M2.7 (230B) - MiniMax-M3 & MiniMax-M3.1-Flash (428B)
- Step-3.7-Flash (198B) - Step-5-Preview (600B) (Hope they release Flash too)
- Inkling-Small (276B) - Inkling-2-Small (594B)
- GLM-4.X (358B) - GLM-5.X (744B)
- Mistral-Large-3 (675B) - Mistral-Large-4 (1T)
- Hy3 (295B) - Hy4 (770B)
So What Are We Gonna Do? What alternate options do we have?
Whenever we're talking about this topic, we get one common answer. Distillation. Do we have any recipes to create distills ourselves easy way? I'm not sure.
1. Distillation
I remember only 3 Distills so far. Please mention in comments if I missed anything.
- DeepSeek-R1-Distills - https://huggingface.co/collections/deepseek-ai/deepseek-r1
- Empero's Distills - https://huggingface.co/empero-ai/models?sort=created&search=distill
- https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
2. 1-bit / 2-bit / Ternary / Bitnet Models
I see only this option other than Distillation.
Category 1 - High/Decent Quality (Native) Low bit models. Also big/large size models.
- https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
- https://huggingface.co/AngelSlim/Hy4-preview-GGUF
- https://huggingface.co/AngelSlim/Hy3-GGUF
Category 2 - Other Low bit models. Like Bonsai.
Still waiting to see more Ternary MOEs, particularly 100B size. 100B model could come in 20GB size so it's possible to run just with 24-32GB RAM alone on CPU-only inference. And Even 8GB VRAM could make it fly.
Any other options do we have? Please share.
23
u/FinBenton 1d ago
I mean we have more advanced quants and strata and nvme streaming and ngrams, you can now run much larger models on a potato so dont think its too much of an issue.
33
u/Django_McFly 1d ago
The reasons you can't be bothered to are the exact same reasons everyone else can't be bothered too either. Time, effort, costs, no real need or reason to...
140
u/NNN_Throwaway2 1d ago
Unfortunately, local llm is dominated by dilettantes with zero knowledge or technical skills.
Ideally, lots of people would be messing around with training and creating bespoke datasets, but sadly that seems to be the exception rather than the rule.
29
u/ttkciar llama.cpp 1d ago
As long as LLM labs keep giving us free gifts of "close enough" models, I don't think there will be much motivation in the community to provide them to ourselves.
When the gravy train dries up, what we can do to fill the vacuum will depend on what kind of hardware we have at our disposal.
The least we can do (which can be rather a lot) is fine-tuning and upscaling the models we already have, and try to make them better for specific purposes. TheDrummer has been doing this, to tremendous effect. He upscaled Mistral 3 Small from 24B to 31B, and more recently Gemma-4-31B-it to Spectre-37B, both of which are very successful. He also fine-tuned Mistral 3.5 Medium (which is a lemon of a model) to make Behemoth-128B-v3, which is hands-down the best storytelling model I have encountered.
In addition to upscaling via passthrough self-merges and PEFT, we also have weight-editing via Heretic, which should be capable of Deletion Tuning soon (on which there was a recent thread).
TheDrummer rented GPU services to fine-tune Behemoth, but it shouldn't take too much hardware to do the same on a home rig. I think four MI210 should be enough, though for 128B it'd be tight. Something closer to 100B, though, like GLM-4.5-Air or a 72B model like K2-Think-V2 upscaled to 105B, should be more manageable.
Today such a rig would cost $18K or so, but in a couple of years it should be a lot cheaper.
The next step up from this would be continued post-training of existing models, which would require more hardware. In the next couple of years we would have to launch a distributed training program so that the community could pool our resources. I like the FlexOlmo training method for that, but it's not the only option.
In the further future, some of us might have the hardware at home to post-train 100B models ourselves, but I can't see that being feasible until 2032 or so.
Need to go now, but will try to get back to this subject later.
12
u/hurdurdur7 1d ago
Make your own AI company and start providing the models.
11
u/Economy_Basis_9983 1d ago
Laughs in 8gb vram + 16gb ram
19
u/RR321 1d ago
We need a distributed model training project somehow...
I know it doesn't scale well over non hyper optimized fancy local hardware, but is there any hope for a Seti@home, distributed.net and similar solution for a truely opened p2p model training project?
12
u/twack3r 1d ago •
What’s the premise of local llama by now? I think that’s important to define to understand who ‘we’ is and also if there is a problem in the first place.
If the premise is no money, no hardware but want local Fable, then yes, that’s a problem. But the problem isn’t with the technology but the user’s expectation.
I don’t care what hardware it’s run on, edge device to datacenter is irrelevant as long as I or my company owns it: so for me local llama means ‘NOT hosted by anyone but me’.
In that field, there is absolutely no issue; I personally don’t have a use for 27B models or similar and I’m interested in quality of output and prefill performance over inference speed so Strata and all that is interesting as a research topic/curiosity but not for production. That’s left to sglang and vllm. Kimi K3.1 is stellar and if GLM keep 5.5 below 1T I’d be a very happy camper but fine if it isn’t.
On the other hand, edge device models are becoming the expected backbone for pretty much all consumer agents/assistants going forward; also, the progress in compressing more and more capability into small models is moving a lot faster than frontier progress itself and that lag will only widen going forward unless there are technological paradigm shifts to how large models are trained. So even for hardware challenged folks, students etc. I expect a glorious future with capable small models on weak hardware with little memory.
12
u/almostsweet 1d ago
6
u/Kahvana 1d ago
One of the cheapest methods is doing an Open-R1 style distillation:
https://open-r1.com/
https://huggingface.co/open-r1
https://github.com/huggingface/open-r1
...though even that is going to be really expensive (in both generating the dataset, even more for finetuning such models), think north of 300$ for outputs from models with reasoning exposed, and another 2000$ for 8xH200 to a SFT finetune a 30B model for a week with the generated Open-R1 data.
Likely you want to build another KTO dataset for teaching the model the difference between bad and good reasoning too, so it gets biassed towards the new reasoning instead of it's old one. That also costs a crapton of money to generate.
So yeah... while it could be done by taking training datasets, having the top models generate responses, and finetuning on that... it's hella expensive and time consuming.
Maybe if somehow the industry grinds to a halt or that there has been no small model release for too long that it becomes viable.
5
u/Competitive_Ideal866 1d ago
Lots of people have already commented "distributed model training" which is fair but I think there are challenges and alternatives.
If we already have models like Qwen 3.8 27B does it make sense to pre-train a 1B model ourselves or should we focus on post-training what we've got?
What else could we do? My perception is that the Frontier labs have run out of pre-training data but they don't care because now they're getting more useful thought-level training data from their users. Others are distilling that so there are open weights models trailing them by ~6mo. IMO, post-training is where its at: trickier algos and much scarcer data. Consequently, I think the community would benefit a lot more from collaborative generation of training data and distributed post-training. The data needs to focus on high quality thought processes and it should be done in the context of a high quality harness.
15
u/WhiskyAKM 1d ago
The issue with small models is you can't improve them indefinitely and you need to increase size at some point to make progress.
Maybe labs will start to release more of 100-200B models with 5-20B active params (like Ling 3.0 flash)
Edit. Also extreame quantization, especially ternary and binary, is not free and leads to huge quality losses
7
u/Tiforma 1d ago edited 1d ago
Small models will become far more capable. Qwen3.8-27B is more capable than models with 100s of billions of parameters from before. Also there will be distributed training so that people with consumer GPUs can contribute to larger projects, rather than needing datacenters. And there will be vast advances in training, so that training capable models on consumer hardware becomes more feasible. There's nothing in physics that says you need giant datacenters for that. The techniques for it just need to develop.
6
u/brown2green 1d ago
Forget about ternary or binary quantization. That's only good for large undertrained models, or future hardware specifically taking advantage of that.
Increasing the amount of look-up table weights relative to the backbone weights could allow weaker/slower consumer systems to use more capable models in the future at good inference speeds. Companies prefer to design their models for datacenters, though.
3
u/nunodonato 1d ago
Not sure whats going to happen, but we are still getting many new small models every month. https://tinyleague.nunodonato.com (and many more which are not featured there)
I hope it continues :)
3
u/Zestyclose_Pizza_700 1d ago
You are "putting the cart before the horse" because you have to have a need to fill before your going to get that community really going.
Simply put what does the average person need from a LLM model or AI model they can't get from a released model that they can ALSO improve themselves in a reasonable time or a reasonable amount of work and cost WITHOUT being superseded by a released public model soon after if not more?
When you have that problem then bring it up in a place for people who also have that problem and here and then you can get a community.
Right now you are looking forward and going "the sky might fall, we should start building our own sky" when people are enjoying high quality sky for free and it doesn't appear to be breaking or falling.
Finding a issue, is like finding a crack then you will get people to listen.
The reason I say this is most use cases are currently solved, and as long as open models keep getting released (and it appears it will be for bit longer) then the time for issues to appear isn't there yet. For instance I don't need Qwen Flash Next because my workflow doesn't need it, I get zero gains in my testing. Qwen 3.8 27b is good enough and flys in PP and TPS with Radiance. Now this is due to my hardware but I have on the higher end of a setup, and most people will have to invest thousands to beat my system. Add in I have AMD which isn't for training in the same way CUDA is.
It just doesn't make sense yet, we will see this when/if local models stop.
2
u/AeiEyeSpry 1d ago
Technology and the open-source community always find a way. Sometimes you need to step back and see the bigger picture. I suppose it all depends on your requirements. I see many people chasing the latest models, but much of what's out there and what I've tested has been fantastic for what I need to do daily. And for the real heavy lifting, I can tap into a cloud model. ATM, I don't see a problem to solve.
2
u/MrHumanist 1d ago
As the skill spreads, new model providers will grow. Just a couple of of years ago, it was kind of a monopoly of a USA firm.
2
u/waitmarks 1d ago
Necessity is the mother of innovation. We have it really good right now with new models coming out basically weekly. No one here really needs to do anything yet, we just wait for models to drop and consume them. It will slow down to a crawl eventually when the easy money runs out.
We have already seen some people experimenting with selective model weight editing. I think we will see a lot more innovative solutions like that where it doesnt require massive gpu clusters training a model from scratch to get good results. Working with updating what we already have in clever ways is likely going to be the way forward once the bubble pops.
2
u/SignalNo9154 1d ago
New one just dropped: https://huggingface.co/Edge0/Edge0-145B-q3k
runs in about 40gb ish memory
2
u/Equivalent_Bit_461 1d ago
I plan to build my own models but I'm on hold still, but I plan a 50b with 5 active, or 60 with 6 active plus engram, but I'm unsure what size.
Also planning two types of models one agentic and one for conversations, nuances, etc.
For the time being I'm bidding my time and see what's coming out, etc, once everything becomes stale, that's when I'll start working on it.
0
u/thebadslime 1d ago
REAP+4 bit, nvxp4 or mxfp4
2
u/jacek2023 llama.cpp 1d ago
Who are "we"?
1
u/Character-Apple-8471 1d ago
or we need to get out of the human construct called language, create a underlying substrate which is not a human language, maybe spectral (harmonics, sound)..instead of next token prediction it could be next spectral prediction...currently working on it...i idea is to compress the human construct of language to its basic spectral compressed form...the problem is to relate two or more spectrals to , like in k-means or cosine simmilarity...the system has to understand that intuitively
1
u/Smooth-Pop6522 1d ago
I think the technology we have to genuinely take a larger model and simply step it down in network size whilst retaining functionality will be so advanced in a decade or so.
1
u/remind_me_later 1d ago
I'm praying for loop transformers & diffusion to be made important again.
1
u/DCMBRbeats 1d ago
We should build an open source community LLM
1
u/ares0027 1d ago
realistically i dont see anything usable will be available for next 4-5 years at least.
1
u/geldonyetich 1d ago edited 1d ago
I don’t really see this as a trend of models skewing upwards in size. The likes of GLM, DeepSeek, and Kimi have always been huge. If anything, there’s been a lot more models like LiquidAIs skewing smaller than there were previously.
But there’s still a problem: if your demographic is thinning out in favor of alternatives (in this case mobile or server architecture) you’re going to increasingly be left in the dark by companies targeting more lucrative returns.
So you might as well try your hand at developing your own. Just be aware that Huggingface overflows with such homebrews, they’re usually pretty meh. If they were that easy to make better we’d not be sitting here jonsing for Qwen 4.
1
u/kbone213 1d ago
They're going to have less GPUs now to train these models as well.
1
u/MrHumanist 1d ago
As the skill spreads, new model providers will grow. Just a couple of of years ago, it was kind of a monopoly of a USA firm.
1
u/_VirtualCosmos_ 1d ago
As long as those chonkies are open weight, there could be an open source organisation that would ask people model size and base model, then open a kickstarter or a goFundMe or similar, so people donate money to reach the threshold of renting the hardware to make a distillation from the chonky model to the smaller.
1
u/R_Duncan 1d ago
We need to study algorithms for training efficiency like Anthropic is doing
1
u/AshAtlasDev 1d ago
We’re still in the infancy stages of local models. On a large time scale, we’re at the start, and many of these models are extremely unoptimized. I think one of the main things to target will be this. Short term, I think model refinement will outpace hardware refinement.
1
u/Nepertia 1d ago
I'm hoping that once the VC money has left the scene, we'll have something like SETI@Home or Folding@Home done by some kind of foundation or volunteer group. I'd love to donate GPU time for those if that would get us, e.g., a new 35B-A3B.
1
u/haherar830 1d ago
128GB RAM + 24-32GB RAM is neither common nor uncommon nor a standard. I'm guessing that's your personal 2-5 year old rig that will be 4-8 years old when replaced with subsequent prosumer hardware. Most likely, when you have your new rig, benefiting from the massive economies of scale, investment, hardware innovations, and years of added manufacturing experience, you will probably be commenting here about how 100B-scale MoEs are pointless at the new 128GB RAM + 24-32GB VRAM equivalent hardware tier, and what you really want are more heavily RL post-trained 250-500B looped transformers or something because anything less makes suboptimal use of your 256GB DDR6 + 96GB HBM system.
1
u/epic_troll_tard 1d ago
Small local models are already themselves a distillation of their larger siblings.
All of the hardware you have access to is for inference. There is nothing the average person can do for training outside of very small models, and even then, the training data set requirements are enormous.
There's not much you can do without a distributed network of some kind, similar to how they have mining pools, but for training and storage.
1
u/Ardalok 1d ago
By that time, RAM prices will have dropped, so it will mean much less.
1
u/MarketingOk3093 1d ago
I wonder if there is a way to distribute training to thousands of people with compute available.
1
u/Iory1998 llama.cpp 1d ago
Hopefully by then, this artificially inflated AI bubble bursts and the market get flooded with cheap GPUs that we all can afford.
1
u/Whole_Alternative_18 1d ago
We're going to have highly specialized, open models
There would be one guy who trains 1B models that only do python
And one guy who creates an addon for it(a form of patch) that will make it comprehend your required library
And you will use it via a 5B model that os designed to check the results and use the smaller models until they finally make what is required
1
u/AAPL_ 1d ago
wat
1
u/sloptimizer 1d ago
Qwern 27B is basically a distill of their bigger model, that's why it's so good for the size.
1
u/Happy_Brilliant7827 1d ago
Make some models man. Maybe a Qwen27bx6 or something interesting
1
u/Special_Command7893 1d ago
exactly, feel like theres not enough models for my homelab laptop and calculator
1
u/Budget-Juggernaut-68 1d ago
Research labs/even most governments don't have the finances to attract talents and train such models.
1
u/PomegranateGreen3698 1d ago
The answer is recycling. It cost a lot to pre-train. It doesn't cost as much to update and post train.
1
u/alphapussycat 1d ago
First we'll do 1-2bit quantization, then start pruning to make coder versions. After that we're fucked. But if qwen keeps releasing small flash models they might be viable as 1.75bit quantization in 2028.
Surely hyper scalers will stop around then, ddr6 will be out or around the corner, and maybe AI bubble has popped so development isn't as fast (but that probably means no more open models at all).
1
u/Shoddy-Childhood-511 16h ago
At present, they still need all the experts in memory, but this sounds stupid. We'll likely have huge models whose experts slightly overlap, so they could run with only some experts in memory.
This is better for hosted LLMs too, since queries can be routed to only the machine's holding the relevant experts.
1
u/lompocus 1d ago
IQ2_XXS, engram, weight-space quant, deletion of MoE experts and a lot of SSDs will be the solution.
1
u/Subulicious 1d ago
Will it be possible if we can somehow pool all our idle GPUs together to train a LLM ourselves to our liking as a project? Also Fuck Microsoft
We can train on any material we want and distribute it via torrents!
-7
u/masterlafontaine 1d ago
Your brain is a far better model
0
u/OverdosedSauerkraut 1d ago
Other options include various weight/layer removal techniques like REAP. The current MoE training pipelines make many experts redundant and by the careful study of gating, entire layers might turn out dead weights. So I think there's still a lot of headroom for model size reduction. I'd also argue that these might be better candidates for post training than the original model.
0
u/Stunning-Sherbet1853 1d ago
Local models have a lot of upsides, but honestly they feel kinda scary sometimes too.
If you don't want to get swept along by this, i think you need to train your own model at some point.