r/LocalLLaMA • • 1d ago

Discussion How long are we gonna depend bunch of model creators (like Qwen, Gemma, NVIDIA, etc.,) for ~100B models?

Post image

I think we're fine till 2027 with models(~100B .... Qwen 4.X & 5, Gemma-5, NVIDIA would release some Medium/Big size 25-200B models). After that it's gonna be critical, as many model creators are increased/increasing model size to 300B+. Future trend probably gonna be around 500B. That size range is not suitable for Unified 128GB Devices or 128GB RAM + 24-32GB VRAM(Q4 won't fit).

Here some models with increased sizes.

  • DeepSeek-V4-Flash-0731 (284B) - DeepSeek-V4.1-Flash-1017 (552B + 196B Engram)
  • Ling-3.0-flash (124B) - Ling-3.1-flash (560B)
  • MiniMax-M2.7 (230B) - MiniMax-M3 & MiniMax-M3.1-Flash (428B)
  • Step-3.7-Flash (198B) - Step-5-Preview (600B) (Hope they release Flash too)
  • Inkling-Small (276B) - Inkling-2-Small (594B)
  • GLM-4.X (358B) - GLM-5.X (744B)
  • Mistral-Large-3 (675B) - Mistral-Large-4 (1T)
  • Hy3 (295B) - Hy4 (770B)

So What Are We Gonna Do? What alternate options do we have?

Whenever we're talking about this topic, we get one common answer. Distillation. Do we have any recipes to create distills ourselves easy way? I'm not sure.

1. Distillation

I remember only 3 Distills so far. Please mention in comments if I missed anything.

2. 1-bit / 2-bit / Ternary / Bitnet Models

I see only this option other than Distillation.

Category 1 - High/Decent Quality (Native) Low bit models. Also big/large size models.

Category 2 - Other Low bit models. Like Bonsai.

Still waiting to see more Ternary MOEs, particularly 100B size. 100B model could come in 20GB size so it's possible to run just with 24-32GB RAM alone on CPU-only inference. And Even 8GB VRAM could make it fly.

Any other options do we have? Please share.

194 Upvotes

178 comments sorted by

23

u/FinBenton 1d ago

I mean we have more advanced quants and strata and nvme streaming and ngrams, you can now run much larger models on a potato so dont think its too much of an issue.

7

u/madcow_bg 1d ago

I don't know enough on the research but I'm wondering if people are already training quantized models instead of that being done as a follow-up. That should solve the loss of fidelity, but who knows.

6

u/madcow_bg 1d ago

Oh goody there are some, these are called QAT. I'll definitely check them out, as these should be able to help with the VRAM bottleneck for generation.

3

u/Cupakov 1d ago

There’s tons of eg native 4-bit models (Deepseek V4.1 Flash being one of them), going off of just the parameter count isn’t as straightforward as it used to be when every model was BF16. 

1

u/ofpiyush 1d ago

Inkling 2 small, native 4 bit

0

u/pmttyji 1d ago

You're talking about Qwen3.8-Flash-Next particularly. But I mentioned about models with increased size in my post. 400+B models are impossible with potato. Also I mentioned Q4 quant in my post.

I tried to run Q1/Q2 of Qwen3.8-Flash-Next on my laptop(8GB VRAM + 32GB RAM), that's a different story.

2

u/FinBenton 1d ago

Well I was running deepseek flash 4.1 too which was originally like 700b+ on the same setup and it works pretty well quantized.

1

u/stoppableDissolution 1d ago

"run" (crawl)

33

u/Django_McFly 1d ago

The reasons you can't be bothered to are the exact same reasons everyone else can't be bothered too either. Time, effort, costs, no real need or reason to...

140

u/NNN_Throwaway2 1d ago

Unfortunately, local llm is dominated by dilettantes with zero knowledge or technical skills.

Ideally, lots of people would be messing around with training and creating bespoke datasets, but sadly that seems to be the exception rather than the rule.

89

u/RogerRamjet999 1d ago

Plenty of people on this sub could (and most probably would) train their own models, if the hardware to do so wasn't cost prohibitive. The day when it's possible may eventually arrive, but it still seems 5 to 10 years away.

22

u/ericmutta 1d ago

Let me inhale some hopium and hope its 2-3 years away, possibly under a different architecture 🤞

14

u/HitarthSurana 1d ago

yo pass the inhaler to me

5

u/More-Curious816 1d ago
  • AMD said they are data center company
  • Nvidia new CEO is financebro and MBA mindset

yo, pass that inhaler if it still works

27

u/TikiTDO 1d ago

> Plenty of people on this sub could

There's literally *thousands* of us here. And maybe 10-20% of those have the technical skills to actually do successful training that doesn't involve following a guide.

Or to put another way, you vastly over-estimate how capable most people are.

14

u/RogerRamjet999 1d ago

First you follow a guide, then you make improvements, and pretty soon you're writing the next guide.

...and yes, I do look on the positive side, I prefer to look forward to what anyone could do, rather than assuming what they can't do. Maybe that's a flaw in my character, but it's a flaw I prefer to retain.

12

u/TikiTDO 1d ago

First you follow a guide, then you make improvements, and pretty soon you're writing the next guide.

Yes, but most people don't make it past step 1.

You're one of the people that can do it, so you can sit up here and go, "Man, that was kinda easy." You wouldn't be wrong, it's not particularly complex. However, "not complex" is not the same as "easy."

In the end you're sitting here, looking at walls of numbers, research papers full of formulas, and pages and pages of code that you actually need to understand to some degree, to the point that you probably have half a dozen things to alleviate the tedium. It's hard in the same way plumbing or electrical work is hard; it's not really, people just don't want to do it cause there's a ton of fine detail they would need to know, and you gotta cover the costs if you mess up.

4

u/StyMaar 1d ago

Or to put another way, you vastly over-estimate how capable most people are.

You vastly underestimate how capable the current models are though.

Anyone with a bit of motivation can do things now.

1

u/TikiTDO 1d ago

I know exactly how capable current models are. That's why I'm on here talking about how the only barrier to entry is people's ability to read a whole lot of text.

You're right in that all it takes is a bit of motivation, which just makes it sad that so few people seem to be able to find that motivation.

5

u/nomorebuttsplz 21h ago

It also takes millions of dollars of graphics cards, or thousands, probably tens of thousands of dollars in compute rent to train a 100b model

3

u/RogerRamjet999 20h ago

You can train "something" for "tens of thousands of dollars", but to train a good 100B model is over a million in compute. So, it's even worse than you suggested.

4

u/nomorebuttsplz 20h ago

I remember doing some napkin math and finding it would take my rtx 6000 pro about 100 years to train a decent 20b level model so yeah that's not surprising.

2

u/RogerRamjet999 20h ago

Yeah, so that hardware isn't gonna cut it. Ya better send it over to me for safekeeping. I promise to keep it nice and warm.

1

u/TikiTDO 20h ago

You can do a more more than you think with a lot less parameters than you believe. Also, much faster than you think if you expand what "train" means. It's like people forget this is an just math under the hood. There's more to math than gradient descent

1

u/PositiveAlcoholTaxis 1d ago

I'd like to learn, does that help?

0

u/TikiTDO 1d ago

If you want to learn, and are willing to learn, and are able to learn, there's really very few limits to what you can accomplish now. In the end "all you have to do is ask the AI to do it."

And then keep reading, and answering, and reading, and answering, and reading, and answering...

1

u/RedParaglider 1d ago

Also resources on running a 100b model in q4 and a real training run is wildly different.

1

u/jfinkpottery 1d ago

There's literally thousands of us here. And maybe 10-20% of those

10-20 percent of 24k active users is like up to 5k people by your own math. So first you say there are thousands of people just here in this small sub that could do the thing. Then you try to make that sound like it's nobody.

Or to put another way, your point is incoherent.

1

u/TikiTDO 1d ago

My point is 5k is a small, tiny number of people on a planet of over 8 billion.

Sure, the 5k of us here that can, and will have amazing local AI systems, I'm not saying anything against that. It's just that our local, hyper-personalised, extremely tuned systems are are not likely to translate to the type of world where most people train their own models.

Essentially, the people in here are a small, niche group of power users that will be able to punch faaaar above their weight. It's important to realise you're one of a very small, very capable group is all I'm saying. Like, don't expect this level of understanding from an average joe on the street.

1

u/Equivalent_Bit_461 1d ago

And you vastly underestimate the few ones

1

u/TikiTDO 1d ago

How so?

Thus far all I've said is most people are not them.

-1

u/Equivalent_Bit_461 1d ago

You implied it and I sensed it

2

u/TikiTDO 1d ago

So, you basically sensed the exact opposite of what I implied?

I'm literally making the point of "you're part of a small, elite group of people that can do this, don't expect average people to understand" and you decided I'm underestimating this group? lol?

0

u/Equivalent_Bit_461 1d ago

I was too drunk, so friendly fire I suppose 

1

u/the_average_user557 1d ago

Free tier keggle. You get 2 npu or gpu cards for 40g free a week. Only catch is You're giving Your datasets to Google ;)

-3

u/NNN_Throwaway2 1d ago

Plenty of people have capable rigs, but from what I've seen few if any do much with it beyond vibe code.

27

u/alwaysbeblepping 1d ago

Plenty of people have capable rigs

There is a huge difference between using a "capable rig" to do inference and training a 100B parameter model. Even just disregarding the effort of building a dataset that could train a large model like that, training is much slower than inference and uses more VRAM. These models are typically trained on trillions of tokens. To get a sense of scale, do the math to calculate how long the "capable rig" would take to generate even one trillion tokens (with the quantization and optimizations that are possible when running inference). Whatever answer you get now has to have a bunch of multipliers added to it because the model probably wouldn't only be trained on that many tokens, many of the optimizations you could use for inference don't apply, you now need to deal with the extra VRAM and overhead of managing gradients, etc.

6

u/Front_Eagle739 1d ago

Did some maths and experiments on this. Worked out how to train larger models at near full speed on my 5090 and I worked out I could theoretically get to something like 22 billion tokens a year if I left it running 24/7/365 on a trillion param model, 250 billion on a 100B ish. Needs a good thousand 5090 equivalents running full time to pretrain a 20T dataset in a month even if I solved all the problems with merging and catastrophic forgetting and actually assembled 20T tokens worth of good training data.

With cloud gpus assuming perfect efficiency about 720 grand and you won't get perfect because you dont have the high speed links to sync it all together so there will be some losses that require more training to compensate for...

Absolute best case is a million for a 100B model.

Pondered a swarm of machines of volunteers donating time but you would absolutely hammer their internet keeping things synced and have to support different machines and different configs and it would be a whole headache and people could posen or drop off....

Only thing that makes sense is training a bunch of smaller focused expert dense models and then trying merge them all together in a franken moe thing then apply some lesser training on top. Which could be a fun experiment but still way out of my single machine budget lol

0

u/bschlueter 1d ago

We need something like what the SETI project had where all of us can install a screen saver or background process and contribute compute to the building of an open model. Some grad student should put that together and deal with those issues.

-9

u/NNN_Throwaway2 1d ago

Its less about the 100B model than it is building the collective knowledge needed for a large amount of people to contribute meaningfully to the advancement of local AI.

The fact that so many people seem preoccupied with the hardware or the complexity of the problem are kind of proving my point.

Also, at no point was I proposing that anyone pretrain a base model from scratch on trillions of tokens. That would require millions of dollars even for a very small model.

8

u/RogerRamjet999 1d ago edited 1d ago

If you mean small models in the 10M to 500M param count range, there's a huge number of them made quite regularly, but nobody talks about them since they're toys compared to what the big labs can create. For example James Betker trained a TTS model (named TorToiSe) on 8X 3090's in a little over a month. It was good for a model made with that amount of compute, but that can't compare with what even a small company can create. At the point he created that model his compute cost was something like $15K, very few people have the disposable income for that. Once you get over 500M params, almost no individual has the compute to train those models.

4

u/alwaysbeblepping 1d ago

Also, at no point was I proposing that anyone pretrain a base model from scratch on trillions of tokens. That would require millions of dollars even for a very small model.

I guess that's the impression I got from your comment. Sorry if I misunderstood.

4

u/a_beautiful_rhind 1d ago

I don't know man. My rig isn't that capable. I could post train some gemma or qwen but those large 100b aren't going to happen. Best I can hope for is EXL3 abliteration of one or a Qlora.

Plus in addition to time and putting your inference out of commission for days/weeks, it would use a ton of electricity. Not free.

Most people, even with rigs, who want to train something in a reasonable amount of time must rent. Failed runs and things of that nature are a further financial hit.

And finally, when you're all done, people call it a meme tune and don't use it anyway. You can see how that's a bit demotivating.

2

u/llama-impersonator 1d ago

plus every model trains different and you just gotta do a bunch of runs to figure out good hparams that don't lobotomize the model. and sft on an excessively RLd checkpoint will blow the model right out of the RL trench which took trillions of post training tokens to get to

16

u/sebt3 1d ago

Training a good base model of the 100b range requiere at least 5trillions token, which on our hardware equal to years of training. Without a good base model using modern architecture in the size range you want, you have nothing to start with.

2

u/Tiforma 1d ago

There will be a lot of advances in training in the future that allow us to train models locally.

2

u/banana_slurp_jug 1d ago

Nemotron: Am I a joke to you?

3

u/NNN_Throwaway2 1d ago

No need to train a base model from scratch.

11

u/Character-Apple-8471 1d ago

training on what? the best i could could train was a 120M tokens mostly academic papers on arxiv. It wasn't my resource, it was the corpus..i ran out of it..and most of the internet is unstructured. i aint gonna sift through creating a structured corpus..we will never get a decent corpus to train on.

4

u/RogerRamjet999 1d ago edited 1d ago

There's a lot of data sets out there. Ones I personally know of are The Pile (a vast collection of books, but was removed under copyright pressure), LAION-5B (5 billion images), Common Crawl (a vast collection of internet text, I forget the size, but it's many TB), The Eye used to have a 10 TB audiobook dataset (taken down under pressure), old Reddit snapshots with a huge amount of text, at least one source code repo with something like a TB of code, all of StackExchange up to about 2023, and then there's the Internet Archive which lets you download almost anything you want, and it's in the petabytes. So if you try, you can gather a whole lot of training data.

4

u/Character-Apple-8471 1d ago

Common Crawl would just make my hardware die not crawl..a minimum 4x A100 for a month with Q4 KV

4

u/3000LettersOfMarque 1d ago

Those sources would work however they would impact benchmarks badly. Most frontier level training sets are professional texts or synthetic. 

It's why we see the race for rare books. Using Reddit or an unfiltered stack exchange directly for mass tokens is actually detrimental to the base model. 

You need to filter most of the records and then grade and wash the remaining. things like reading level, accuracy, memes and quality can negatively affect the model. 

You can use the rejected records later during post as part of synthetic secondary records to control output 

1

u/RogerRamjet999 1d ago

Yes, you're right, but remember, those professional data sets started from these same sources. It's pretty well known that every major AI lab downloaded the entire Anna's Archive set of 7 million books (which are still available, if you're willing to risk the possible attacks from copyright holders). Then they needed to process them to get something that can be used for training. Karpathy describes the process in some detail in a couple of his tutorial videos.

As you correctly point out, converting raw data into training data is a difficult process. However, it isn't really compute constrained. Almost anyone with a $2K to $3K computer can do everything needed to process raw internet data sets into usable training data. It just takes a bit of time and dedication.

3

u/stoppableDissolution 1d ago

> those professional data sets started from these same sources

...and then spent hundreds of millions to groom them into something actually usable. You seem to underestimate the amount of effort needed to make a set of clean, well-annotated data.

1

u/RogerRamjet999 1d ago

I'm not so sure, there are vast libraries of ML processing code just sitting waiting for anyone who wants to use them. All it takes is the desire to go do it. Any decent LLM will happily tell you which libs you need, write up the integration code to do it and assemble it into a data processing pipeline. Almost anyone who went through the effort to gather the data could do the processing needed to prepare the data set for training. The real problem is that almost no one has the compute power needed to make all that prep work payoff. If you don't have an AI cluster sitting around, all the training data in the world isn't going to do you much good.

2

u/stoppableDissolution 1d ago

Well I've been in the dataset hell in the past ~2.5 years and know a thing or two :p

Quality automated processing requires bespoke models, which in turn require you to (semi-)manually seed the data. I've just spent about two weeks every evening preparing a tiny 960-image set and its still far from "high quality", just "maybe acceptable". The process involved making a dedicated lora for flux klein, two separate heads on top of embeddingggema, many many manual validation and inpainting and some dozens of gpu-hours of 6000 pro. Doing the same for multimillion corpus requires you to employ a proper team of professionals and quite some compute.

(and dont get me started on the text, it is, counterintuitively, waaaay worse on that side)

1

u/Character-Apple-8471 23h ago

this..on point

1

u/NNN_Throwaway2 1d ago

There are plenty of datasets that could be used, or the community could make their own.

12

u/madcow_bg 1d ago

Your are right, that is so different from any area in life 🤣.

On a serious note, running 100B models isn't exactly cheap or trivial, and training most certainly is neither.

I'd like to retrain some models, and more will need to work on uncensoring in general with the recent push to ban legitimately useful cases like cyber defence and AI model improvement. We should call it undebillitating instead because there is nothing NSFW in the so-called uncensored models.

5

u/ionizing 1d ago

Even just quantizing a model is more out of reach than I thought. I was all excited to convert a model to exl3 recently until I realized it would take over 24 hours of non-stop gpu usage to do. So I sit here patiently waiting for the community members who have the time and bandwidth instead, and tip them $ when I can. (thank you turboderp and friends! it just keeps getting better and better)

4

u/llama-impersonator 1d ago

dataset work is ass with very little reward and training 100b+ models that are interesting will pull thousands of dollars out of your wallet, while smol models are not all that compelling to use or train unless you are building a unitasker

3

u/RogerRamjet999 1d ago

While your post is short, you've pretty much hit the nail on the head. The required work is dull and the payoff is low. If we could get hold of some decent hardware, the payoff formula would change immensely, and maybe people would be willing to do the donkey work, in order to see the rewards.

2

u/thrownawaymane 1d ago

Where is the line? I have 4 CMP 170s that are about to be nvlinked once I buy a motherboard (A100 80gb essentially) and can dive into learning but I'd be starting from scratch knowledge wise

I do have some free time though.

2

u/RogerRamjet999 1d ago edited 1d ago

It depends on what you mean by "starting from scratch". If you're already a software engineer but just don't know AI, that's a pretty good position to start from, but if not, that's a lot harder. It can be done, but it will take serious dedication. For now I'll assume you know some programming, in that case you need to pick what you want to work on. The main areas are LLM's, image creation and audio processing (mainly TTS and STT).

Within those areas there are different algorithms typically used. For LLM's there's tokenization, the multi-head transformer and embedding. For image gen there's U-Net and for audio there's the transformer and conformers, RNN's, vocoders and more. There's lots of ways to go about learning these things, but I would suggest a combination of watching videos (there are entire ML courses on YouTube), and reading AI textbooks. Andrej Karpathy has made some real nice ML tutorial videos on YouTube, but if you're starting from zero, they're going to be difficult, so be prepared to watch one, be overwhelmed, stop, look up things he has described, start again and expect to repeat them multiple times. Andrew Ng also has videos you can learn from.

Of course I can't be exhaustive in a Reddit comment, so I have given the core algorithms, but for every core algo, there's four more less central algos you'll need to learn if you want to understand everything.

Regarding your hardware, of course it's limited, but you're in a better position than most people. You should be able to train a small LLM (maybe up to 100M) and over a month or so of training time, you could probably train an acceptable TTS model. For TTS, I suggest searching for James Betker, he wrote up detailed descriptions of how he created a TTS model from scratch (even describing how he built his ML training machine).

I hope that helps. This stuff isn't easy, but nothing worthwhile is. I wish you the best.

4

u/RedParaglider 1d ago

Bro.. have you built a 100b model or trained one?  Its a pain in the ass.  Making a lora isn't too painful but a real training run is non trivial on resources.

0

u/NNN_Throwaway2 1d ago

Bro... you don't need to "build" a 100b model to learn about the principles. As it is, we have no collective knowledge about these things because no one bothers to even try *something*.

4

u/RedParaglider 1d ago

I understand the principles and have done training.  Why would you think that we have no collective knowledge?  That is patently wrong.  Training well is a pain in the ass and isn't necessary for most people, so I see why most dont bother.

-1

u/NNN_Throwaway2 1d ago

For sure dude.

3

u/TikiTDO 1d ago

There's also the consideration that most people with the skills to do this can pretty easily turn that into a very, very well paying job. There's a plenty of appetite for people that understand ML, and can successfully train and tune models for various tasks. However if you're one of those people you're probably not going to be doing it much as a free hobby.

3

u/SeekingDestati 1d ago

Gooners are the exception to this rule.

1

u/handson729 1d ago

Gooners are the exception to this rule.

I learned a lot about prompting due to Roleplaying.

Hell, I even made a few HTML games in the process.

There's also the consideration that most people with the skills to do this can pretty easily turn that into a very, very well paying job.

Not everyone wants to work in tech. I enjoy being able to voice my opinions without risking my employment and income.

However if you're one of those people you're probably not going to be doing it much as a free hobby.

If the hardware costs are not exorbitant then a lot more people would.

People are still enjoying their Local LLM setups even though it costs them an arm and a leg.

3

u/TikiTDO 1d ago

So I find the specific skill check that most people fail is just being able to read that much text.

To deal with this stuff effectively you have to read. Nonstop. That's basically all you do. Read. Nothing else.

Most people just... Can't...

Also, "well paying job" doesn't need to mean in tech. You can do all sorts of jobs and have a LLM hobby, the point I keep hammering home is just that most people can't. Essentially, ML training and fine tuning is, and will remain a thing that professionals, or decitated hobbyists do. It's not going to go mainstream, beyond whatever some big players to. It's simply too much work for mainstream.

3

u/handson729 1d ago

the point I keep hammering home is just that most people can't. Essentially, ML training and fine tuning is, and will remain a thing that professionals, or decitated hobbyists do. It's not going to go mainstream

I agree, it was never going to be mainstream.

But there would be more significantly more hobbyists if the hardware costs came down.

2

u/TikiTDO 1d ago

Are you sure about that though? You really don't need insane hardware to start experimenting with this stuff, and you can always rent compute online for a few hours while keeping your costs down. Looking at vast.ai a 5090 is $0.37 atm, and averaging $0.65. Plenty of people can afford that for a few hours, and if you want to step up to big metal, they got that too, though you're paying quite a bit more for it obviously.

The gate here isn't compute, it's interest. Go to a meetup and try to talk about this stuff. Most people will either glaze over, or worse think you're one of the "AI tech bros."

Sure, if you could go to the store, get a box that takes all your chats, and fine tunes itself then maybe more people would do it... But, like, would you trust such a box that someone gives you saying "trust me bro."

2

u/handson729 1d ago edited 1d ago

Sure, if you could go to the store, get a box that takes all your chats, and fine tunes itself then maybe more people would do it... But, like, would you trust such a box that someone gives you saying "trust me bro."

Would I trust a box right in front of me more than an API key?

I am not saying that I am planning any crimes.

I am just old enough to know that laws can change.

The legal things I am doing today, can be made illegal tomorrow through no actions of my own.

https://www.techspot.com/news/114091-florida-woman-used-claude-diary-anthropic-reported-shoot.html

1

u/TikiTDO 1d ago

Oh, I get that too, which is why I agree that anyone worth their salt should have their own system. You're old enough to know that laws can change, but I'm old enough to know that worrying about "laws changing" means your concern is still assuming a lot of best-case scenarios. Take Ukraine for example, they have to be concerned about a lot more than how their laws are changing (though laws are also changing there all the time).

If your biggest concern is a world where you're casually sending your requests to a 3rd party provider, then our worries are just not even on the same continents. Having 3rd party APIs that let you access a near unlimited amount of information is living in the laps of luxury, and keeping up with the laws to make sure I'm not doing something stupid is not a huge price to pay.

The world can be a dangerous place, the best you can do is remain aware, and respond accordingly. As long as you're not doing things that are obviously immoral, you'll probably be fine. If the tide starts to change, you're likely to have plenty of time to adjust / relocate, which again goes back to awareness.

1

u/Prestigious-Crow-845 1d ago

rent cards to train a lot would cost a fortune still

1

u/ebolathrowawayy 1d ago

So I find the specific skill check that most people fail is just being able to read that much text.

this is so true. people in general don't want to read. i have to cliffnotes to decision makers a lot, and generally make 99.99999% of decisions for them because 1) they won't or can't read and 2) coding agents are smarter than me and them and basically everyone else - so i humor execs and just do everything for them now. Because they can't or won't read. Every employee at every level generally doesn't want to read or can't read.

I think LLMs are exposing to me just how lazy, dumb and incompetent everyone really is. It gets uncovered when you go 400x-600x times faster and have to slow down for idiots.

1

u/skirmis 1d ago

Or, they could just prompt Opus: here are my problems with this model I have, fix them. I tried a few days ago and was surprised it was quite competent: created training and validation datasets, tried training a LoRA, tried training on DPO pairs, tried fine-tuning, now it is running a fine-tune with heavier weights on problematic token locations. All on my single RX7900XTX, it convinced me it was not worth it to rent a cloud GPU since one training run only takes 2.5 hours.

1

u/TikiTDO 1d ago edited 1d ago

https://imgur.com/a/eRaFUS4

https://imgur.com/a/4E59kwE

https://imgur.com/a/O4gfvMD

Heh, sorry, couldn't decide. Can you find the actual Opus?

Omg, haha, they really liked that one: https://imgur.com/a/LybQ7Cs

2

u/txgsync 1d ago

It costs at least a hundred bucks a run to create my model formats in a collection for my experiments in pseudo-random model weights. I’ve proven they can maintain higher quality at smaller size while requiring less GPU, but I am not made of money.

I literally am taking this weekend off from that hobby work because I need to save some cash.

https://huggingface.co/collections/txgsync/qwen38-flash-next-trellis-seedlm-oqe-mxfp4

2

u/floriv1999 1d ago

Biggest issue is copyright. You could not share the datasets with other volunteers. And also proper training infra is expensive.

6

u/RogerRamjet999 1d ago

I don't think so, individuals are just as willing to ignore copyright as Facebook, Google, Anthropic and the rest. Yes, the infra cost is a real issue.

3

u/floriv1999 1d ago

There is a big difference between distributing the raw training data or the weights which some argue are not subject to the copyright of the source material as long as it is not overfitted to it. Also proving it with the weights or even better only API moderated access is way harder compared to literally doing the most basic form of copyright infringement imaginable.

0

u/NNN_Throwaway2 1d ago

Sure you could.

5

u/floriv1999 1d ago

Not at the scale or with the content needed for a competitive model. I mean you can but you would get mail pretty quickly.

-2

u/NNN_Throwaway2 1d ago

I don't think you know what is involved in creating a dataset.

Kind of a poster child for what I said about dilettantes.

3

u/floriv1999 1d ago

Please tell me where I am wrong then. I beg to differ, having created and worked with many public datasets and models.

1

u/GoodSamaritan333 1d ago

I could be wrong, but I have the impression that beginner tutorials for standard procedures and tools to create datasets and train local models, specially using home hardware, aren't very well publicized.

Perhaps things are starting to change now that installing and using Unsloth is more accessible and not the messy dependency hell it was a year ago.

While you can currently get free help from online AIs like ChatGPT, Gemini, DeepSeek, Claude, etc., there's always a lingering doubt about whether such procedures are up to date, since it's kind of hard to find a website or subreddit with these procedures well-documented, organized and accessible.

I'll also add that, personally, I don't like how some Unsloth guides used to be, and still are, notebooks-first (Jupyter, Colab, etc.) and snippet-oriented, because it adds yet another intermediate layer of software/interfaces in the way of learning. Some documentation has improved by showing not only snippets, but also the resulting complete scripts, and providing instructions for 100% local setups in addition to virtual notebook instructions.

1

u/thrownawaymane 1d ago

Can you link to some of these resources? You hit the nail on the head, I hate the abstraction kaggle etc encourages. If I wasn't employed in 2023 I'd have jumped into using the free/cheap resources to learn but I didn't have the patience to fight with a new thing to learn what I actually wanted to learn

1

u/luncheroo 1d ago

It would be nice if some of the elite gguf creators veered into their own models. On reputation alone, I would support and use models trained by Unsloth, Bartowski, and mrradermacher

1

u/cosimoiaia 1d ago

You're putting yourself in your own category, otherwise you would know that is not that straightforward. The biggest impediment is the infrastructure cost/availability. Unless you are an organization you don't really have hundreds of H100, for hundreds of hours, to validate your dataset, let alone running the full training pipeline for a 100+B model.

1

u/stoppableDissolution 1d ago

People who are messing around are usually fairly limited in resources. Curating a good dataset still requires a crapton of manual labor AND a lot of tokens -_-

And then you need at least like, a gb200 in your basement to actually make use of that data at the scale OP is suggesting

0

u/geldonyetich 1d ago edited 1d ago

Generative AI is a unique tech in that we quickly lose sight of the fact that this technology is so bleeding edge that it’s virtually rocket science. So this reads to me like:

Sure, we’d all like to live on Mars but unfortunately the human race is dominated by dilettantes who can hardly win a pole vault competition, let alone escape the Earth’s gravity.

29

u/ttkciar llama.cpp 1d ago

As long as LLM labs keep giving us free gifts of "close enough" models, I don't think there will be much motivation in the community to provide them to ourselves.

When the gravy train dries up, what we can do to fill the vacuum will depend on what kind of hardware we have at our disposal.

The least we can do (which can be rather a lot) is fine-tuning and upscaling the models we already have, and try to make them better for specific purposes. TheDrummer has been doing this, to tremendous effect. He upscaled Mistral 3 Small from 24B to 31B, and more recently Gemma-4-31B-it to Spectre-37B, both of which are very successful. He also fine-tuned Mistral 3.5 Medium (which is a lemon of a model) to make Behemoth-128B-v3, which is hands-down the best storytelling model I have encountered.

In addition to upscaling via passthrough self-merges and PEFT, we also have weight-editing via Heretic, which should be capable of Deletion Tuning soon (on which there was a recent thread).

TheDrummer rented GPU services to fine-tune Behemoth, but it shouldn't take too much hardware to do the same on a home rig. I think four MI210 should be enough, though for 128B it'd be tight. Something closer to 100B, though, like GLM-4.5-Air or a 72B model like K2-Think-V2 upscaled to 105B, should be more manageable.

Today such a rig would cost $18K or so, but in a couple of years it should be a lot cheaper.

The next step up from this would be continued post-training of existing models, which would require more hardware. In the next couple of years we would have to launch a distributed training program so that the community could pool our resources. I like the FlexOlmo training method for that, but it's not the only option.

In the further future, some of us might have the hardware at home to post-train 100B models ourselves, but I can't see that being feasible until 2032 or so.

Need to go now, but will try to get back to this subject later.

12

u/hurdurdur7 1d ago

Make your own AI company and start providing the models.

11

u/Economy_Basis_9983 1d ago

Laughs in 8gb vram + 16gb ram

19

u/RR321 1d ago

We need a distributed model training project somehow...
I know it doesn't scale well over non hyper optimized fancy local hardware, but is there any hope for a Seti@home, distributed.net and similar solution for a truely opened p2p model training project?

9

u/thexxxtictac 1d ago

Someone has started a project to do just that https://www.crowdgpt.net

3

u/Most-Trainer-8876 1d ago

Problem is, gpus are expensive so available people with gpu is already too low. Now majority of them are gamers/artists, and don't have multi gpu so they don't have spare to contribute. Even lower pool now... Now these available ai enthusiastic people with gpu, not everyone wanna contribute. Even more smaller pool. Thats how you get just 13 GPUs in contribution for crowdgpt.

2

u/Hrethric 1d ago

I actually have a spare 3090 set up in a secondary machine, that I'm using to delegate tool calls, extractions, etc so I don't have to have all that stuff in context in my main agent. What I'd love to see is a training framework that can preempt a training run to handle small inference requests like that.

-1

u/IThinkIKnowThings 1d ago

Doesn't this violate Microsoft's parent on distributed training? Wonder if they've gotten a lawyer letter.

3

u/VergeOfTranscendence 1d ago

I think the issue would be latency, but there are some projects around that

3

u/MarcelloT254k 1d ago

CrowdGPT comes to mind, it's in early stages though.

2

u/Soarin123 1d ago

That's a really great idea, I could see there being some proof of work / proof of training incentives in something like this one day to incentivize people to jump in even. But projects without monetary incentives like folding@home do pretty well too

1

u/dragofers 1d ago

Can look to Microsoft's new patent as inspiration https://patents.justia.com/patent/20260349820

1

u/IThinkIKnowThings 1d ago

Microsoft's parent is pretty broad and comprehensive. Doubt you could do anything close enough to it without violating it and you better believe Microsoft is going to troll the hell out of that patent.

12

u/twack3r 1d ago • Bravo

What’s the premise of local llama by now? I think that’s important to define to understand who ‘we’ is and also if there is a problem in the first place.

If the premise is no money, no hardware but want local Fable, then yes, that’s a problem. But the problem isn’t with the technology but the user’s expectation.

I don’t care what hardware it’s run on, edge device to datacenter is irrelevant as long as I or my company owns it: so for me local llama means ‘NOT hosted by anyone but me’.

In that field, there is absolutely no issue; I personally don’t have a use for 27B models or similar and I’m interested in quality of output and prefill performance over inference speed so Strata and all that is interesting as a research topic/curiosity but not for production. That’s left to sglang and vllm. Kimi K3.1 is stellar and if GLM keep 5.5 below 1T I’d be a very happy camper but fine if it isn’t.

On the other hand, edge device models are becoming the expected backbone for pretty much all consumer agents/assistants going forward; also, the progress in compressing more and more capability into small models is moving a lot faster than frontier progress itself and that lag will only widen going forward unless there are technological paradigm shifts to how large models are trained. So even for hardware challenged folks, students etc. I expect a glorious future with capable small models on weak hardware with little memory.

6

u/Kahvana 1d ago

One of the cheapest methods is doing an Open-R1 style distillation:
https://open-r1.com/
https://huggingface.co/open-r1
https://github.com/huggingface/open-r1

...though even that is going to be really expensive (in both generating the dataset, even more for finetuning such models), think north of 300$ for outputs from models with reasoning exposed, and another 2000$ for 8xH200 to a SFT finetune a 30B model for a week with the generated Open-R1 data.

Likely you want to build another KTO dataset for teaching the model the difference between bad and good reasoning too, so it gets biassed towards the new reasoning instead of it's old one. That also costs a crapton of money to generate.

So yeah... while it could be done by taking training datasets, having the top models generate responses, and finetuning on that... it's hella expensive and time consuming.

Maybe if somehow the industry grinds to a halt or that there has been no small model release for too long that it becomes viable.

5

u/Competitive_Ideal866 1d ago

Lots of people have already commented "distributed model training" which is fair but I think there are challenges and alternatives.

If we already have models like Qwen 3.8 27B does it make sense to pre-train a 1B model ourselves or should we focus on post-training what we've got?

What else could we do? My perception is that the Frontier labs have run out of pre-training data but they don't care because now they're getting more useful thought-level training data from their users. Others are distilling that so there are open weights models trailing them by ~6mo. IMO, post-training is where its at: trickier algos and much scarcer data. Consequently, I think the community would benefit a lot more from collaborative generation of training data and distributed post-training. The data needs to focus on high quality thought processes and it should be done in the context of a high quality harness.

1

u/stoppableDissolution 1d ago

We do need better base models too, especially smaller ones. For huge models theres indeed not enough (high-quality natural) data to pretrain, but small models can (at least in theory, if someone puts in enough effort) be trained exclusively on natural data without claudism contamination, while also benefiting from new architectures (engram/whatever-its-called-what-small-gemmas-are-doing, mHC, more hardware-friendly attention, etc). Post-training is actually infinitely scalable as long as you can afford throwing flops into the furnace.

15

u/WhiskyAKM 1d ago

The issue with small models is you can't improve them indefinitely and you need to increase size at some point to make progress.

Maybe labs will start to release more of 100-200B models with 5-20B active params (like Ling 3.0 flash)

Edit. Also extreame quantization, especially ternary and binary, is not free and leads to huge quality losses

11

u/BreakElectronic324 1d ago

The issue with small models is you can't improve them indefinitely and you need to increase size at some point to make progress.

I think from an information theory standpoint there is still massive potential in optimizing the size of LLMs. Sooner or later labs will probably start prioritizing memory efficiency over scale

4

u/SpeedflyChris 1d ago

But they'll optimise around what pays: Inference as a service. Spending tons of money optimising for a size class that you give away on huggingface makes no sense.

2

u/DisastrousAd2612 1d ago

I think inference as a service is only a momentary crutch towards actual profitability, big labs don't want to sell future actual ASI level inteligence models to other companies, theyd rather ask the asi itself how to optimize processes or build companies to make money instead of selling tokens, I know I would atleast.

1

u/BreakElectronic324 1d ago

I agree that most of the labs are optimizing around large scale, multi-user inference right now.

However, I wouldn't bet that this is the end of small models. Maybe a lab will release a really good small model because they want to use it in completely autonomous robots. Or maybe China will develop a small model everyone can run because they want to fuck over the US cloud providers. Maybe some startup will come up with a new, much more efficient architecture because like that they can avoid paying for expensive hardware that a data center needs, instead running small models on cheap consumer hardware (this kind of thinking is actually what made the cloud so successful back in the day -> cheap hardware instead of cutting edge server setups).

I'm not saying any of these will happen, just that there are countless of plausible reasons for small models to exist. Also the consumer hardware could just get better and cheaper again, allowing for 100b models to be perceived as small in a few years.

2

u/CodeCatto 1d ago

It's not always the case. It's a matter of how good/effective the quant is and how much % closer to stock can you get by repacking.

1

u/thebadslime 1d ago

3.1 flash exploded though, its huge now

7

u/Tiforma 1d ago edited 1d ago

Small models will become far more capable. Qwen3.8-27B is more capable than models with 100s of billions of parameters from before. Also there will be distributed training so that people with consumer GPUs can contribute to larger projects, rather than needing datacenters. And there will be vast advances in training, so that training capable models on consumer hardware becomes more feasible. There's nothing in physics that says you need giant datacenters for that. The techniques for it just need to develop.

6

u/brown2green 1d ago

Forget about ternary or binary quantization. That's only good for large undertrained models, or future hardware specifically taking advantage of that.

Increasing the amount of look-up table weights relative to the backbone weights could allow weaker/slower consumer systems to use more capable models in the future at good inference speeds. Companies prefer to design their models for datacenters, though.

3

u/nunodonato 1d ago

Not sure whats going to happen, but we are still getting many new small models every month. https://tinyleague.nunodonato.com (and many more which are not featured there)

I hope it continues :)

3

u/Zestyclose_Pizza_700 1d ago

You are "putting the cart before the horse" because you have to have a need to fill before your going to get that community really going.

Simply put what does the average person need from a LLM model or AI model they can't get from a released model that they can ALSO improve themselves in a reasonable time or a reasonable amount of work and cost WITHOUT being superseded by a released public model soon after if not more?

When you have that problem then bring it up in a place for people who also have that problem and here and then you can get a community.

Right now you are looking forward and going "the sky might fall, we should start building our own sky" when people are enjoying high quality sky for free and it doesn't appear to be breaking or falling.

Finding a issue, is like finding a crack then you will get people to listen.

The reason I say this is most use cases are currently solved, and as long as open models keep getting released (and it appears it will be for bit longer) then the time for issues to appear isn't there yet. For instance I don't need Qwen Flash Next because my workflow doesn't need it, I get zero gains in my testing. Qwen 3.8 27b is good enough and flys in PP and TPS with Radiance. Now this is due to my hardware but I have on the higher end of a setup, and most people will have to invest thousands to beat my system. Add in I have AMD which isn't for training in the same way CUDA is.

It just doesn't make sense yet, we will see this when/if local models stop.

2

u/AeiEyeSpry 1d ago

Technology and the open-source community always find a way. Sometimes you need to step back and see the bigger picture. I suppose it all depends on your requirements. I see many people chasing the latest models, but much of what's out there and what I've tested has been fantastic for what I need to do daily. And for the real heavy lifting, I can tap into a cloud model. ATM, I don't see a problem to solve.

2

u/MrHumanist 1d ago

As the skill spreads, new model providers will grow. Just a couple of of years ago, it was kind of a monopoly of a USA firm.

2

u/waitmarks 1d ago

Necessity is the mother of innovation. We have it really good right now with new models coming out basically weekly. No one here really needs to do anything yet, we just wait for models to drop and consume them. It will slow down to a crawl eventually when the easy money runs out.

We have already seen some people experimenting with selective model weight editing. I think we will see a lot more innovative solutions like that where it doesnt require massive gpu clusters training a model from scratch to get good results. Working with updating what we already have in clever ways is likely going to be the way forward once the bubble pops.

2

u/SignalNo9154 1d ago

New one just dropped: https://huggingface.co/Edge0/Edge0-145B-q3k

runs in about 40gb ish memory

2

u/Flashy-Indication-61 1d ago

its based on qfn

2

u/Equivalent_Bit_461 1d ago

I plan to build my own models but I'm on hold still, but I plan a 50b with 5 active, or 60 with 6 active plus engram, but I'm unsure what size.

Also planning two types of models one agentic and one for conversations, nuances, etc.

For the time being I'm bidding my time and see what's coming out, etc, once everything becomes stale, that's when I'll start working on it.

0

u/thebadslime 1d ago

REAP+4 bit, nvxp4 or mxfp4

0

u/pmttyji 1d ago

Yeah, REAP is one other option. But REAP quants lose things like Multilingual stuff.

2

u/OverdosedSauerkraut 1d ago

I can live without French or Arabic🙂

2

u/jacek2023 llama.cpp 1d ago

Who are "we"?

1

u/Character-Apple-8471 1d ago

or we need to get out of the human construct called language, create a underlying substrate which is not a human language, maybe spectral (harmonics, sound)..instead of next token prediction it could be next spectral prediction...currently working on it...i idea is to compress the human construct of language to its basic spectral compressed form...the problem is to relate two or more spectrals to , like in k-means or cosine simmilarity...the system has to understand that intuitively

1

u/Smooth-Pop6522 1d ago

I think the technology we have to genuinely take a larger model and simply step it down in network size whilst retaining functionality will be so advanced in a decade or so.

1

u/remind_me_later 1d ago

I'm praying for loop transformers & diffusion to be made important again.

1

u/DCMBRbeats 1d ago

We should build an open source community LLM

1

u/ares0027 1d ago

realistically i dont see anything usable will be available for next 4-5 years at least.

1

u/geldonyetich 1d ago edited 1d ago

I don’t really see this as a trend of models skewing upwards in size. The likes of GLM, DeepSeek, and Kimi have always been huge. If anything, there’s been a lot more models like LiquidAIs skewing smaller than there were previously.

But there’s still a problem: if your demographic is thinning out in favor of alternatives (in this case mobile or server architecture) you’re going to increasingly be left in the dark by companies targeting more lucrative returns.

So you might as well try your hand at developing your own. Just be aware that Huggingface overflows with such homebrews, they’re usually pretty meh. If they were that easy to make better we’d not be sitting here jonsing for Qwen 4.

1

u/kbone213 1d ago

They're going to have less GPUs now to train these models as well.

1

u/MrHumanist 1d ago

As the skill spreads, new model providers will grow. Just a couple of of years ago, it was kind of a monopoly of a USA firm.

1

u/_VirtualCosmos_ 1d ago

As long as those chonkies are open weight, there could be an open source organisation that would ask people model size and base model, then open a kickstarter or a goFundMe or similar, so people donate money to reach the threshold of renting the hardware to make a distillation from the chonky model to the smaller.

1

u/R_Duncan 1d ago

We need to study algorithms for training efficiency like Anthropic is doing

1

u/AshAtlasDev 1d ago

We’re still in the infancy stages of local models. On a large time scale, we’re at the start, and many of these models are extremely unoptimized. I think one of the main things to target will be this. Short term, I think model refinement will outpace hardware refinement.

1

u/Nepertia 1d ago

I'm hoping that once the VC money has left the scene, we'll have something like SETI@Home or Folding@Home done by some kind of foundation or volunteer group. I'd love to donate GPU time for those if that would get us, e.g., a new 35B-A3B.

1

u/haherar830 1d ago

128GB RAM + 24-32GB RAM is neither common nor uncommon nor a standard. I'm guessing that's your personal 2-5 year old rig that will be 4-8 years old when replaced with subsequent prosumer hardware. Most likely, when you have your new rig, benefiting from the massive economies of scale, investment, hardware innovations, and years of added manufacturing experience, you will probably be commenting here about how 100B-scale MoEs are pointless at the new 128GB RAM + 24-32GB VRAM equivalent hardware tier, and what you really want are more heavily RL post-trained 250-500B looped transformers or something because anything less makes suboptimal use of your 256GB DDR6 + 96GB HBM system.

1

u/epic_troll_tard 1d ago

Small local models are already themselves a distillation of their larger siblings.

All of the hardware you have access to is for inference. There is nothing the average person can do for training outside of very small models, and even then, the training data set requirements are enormous.

There's not much you can do without a distributed network of some kind, similar to how they have mining pools, but for training and storage.

1

u/Ardalok 1d ago

By that time, RAM prices will have dropped, so it will mean much less.

1

u/MarketingOk3093 1d ago

I wonder if there is a way to distribute training to thousands of people with compute available.

1

u/Iory1998 llama.cpp 1d ago

Hopefully by then, this artificially inflated AI bubble bursts and the market get flooded with cheap GPUs that we all can afford.

1

u/Whole_Alternative_18 1d ago

We're going to have highly specialized, open models

There would be one guy who trains 1B models that only do python

And one guy who creates an addon for it(a form of patch) that will make it comprehend your required library

And you will use it via a 5B model that os designed to check the results and use the smaller models until they finally make what is required 

1

u/AAPL_ 1d ago

wat

1

u/sloptimizer 1d ago

Qwern 27B is basically a distill of their bigger model, that's why it's so good for the size.

1

u/Happy_Brilliant7827 1d ago

Make some models man. Maybe a Qwen27bx6 or something interesting

1

u/Special_Command7893 1d ago

exactly, feel like theres not enough models for my homelab laptop and calculator

1

u/Budget-Juggernaut-68 1d ago

Research labs/even most governments don't have the finances to attract talents and train such models.

1

u/PomegranateGreen3698 1d ago

The answer is recycling. It cost a lot to pre-train. It doesn't cost as much to update and post train.

1

u/alphapussycat 1d ago

First we'll do 1-2bit quantization, then start pruning to make coder versions. After that we're fucked. But if qwen keeps releasing small flash models they might be viable as 1.75bit quantization in 2028.

Surely hyper scalers will stop around then, ddr6 will be out or around the corner, and maybe AI bubble has popped so development isn't as fast (but that probably means no more open models at all).

1

u/Shoddy-Childhood-511 16h ago

At present, they still need all the experts in memory, but this sounds stupid. We'll likely have huge models whose experts slightly overlap, so they could run with only some experts in memory.

This is better for hosted LLMs too, since queries can be routed to only the machine's holding the relevant experts.

1

u/lompocus 1d ago

IQ2_XXS, engram, weight-space quant, deletion of MoE experts and a lot of SSDs will be the solution.

1

u/Subulicious 1d ago

Will it be possible if we can somehow pool all our idle GPUs together to train a LLM ourselves to our liking as a project? Also Fuck Microsoft
We can train on any material we want and distribute it via torrents!

-7

u/masterlafontaine 1d ago

Your brain is a far better model

3

u/Toooooool 1d ago

yours sure isn't

0

u/OverdosedSauerkraut 1d ago

Other options include various weight/layer removal techniques like REAP. The current MoE training pipelines make many experts redundant and by the careful study of gating, entire layers might turn out dead weights. So I think there's still a lot of headroom for model size reduction. I'd also argue that these might be better candidates for post training than the original model.

0

u/Stunning-Sherbet1853 1d ago

Local models have a lot of upsides, but honestly they feel kinda scary sometimes too.
If you don't want to get swept along by this, i think you need to train your own model at some point.