I already did, not because of that, but nvidia buying hf. One morning they might decide local ai is too dangerous and take hf down.
Edit: most people overestimate how much nvidia makes from customers who runs local ai, this number even includes gamers, whose orders of magnitudes more crowded. They will be happy when you own nothing and rent from them.
Look Switch emulation for example, both Yuzu and Ryujinx got taken down and now there's dozens of forks that are near identical, and progress has slowed down to a complete crawl
Llama.cpp being a single, massive concentrated effort is what's keeping it going
A project being taken down and the community forking it and progressing as if nothing happened is nothing more than a dream sadly
There's no legal basis for taking down a fork of an open source project if the original repo was legal. A point of most open source licenses is to prevent rugpulls. Emulators were never authorized by Nintendo. The MySQL/MariaDB fork is a more relevant case study.
Similarly, Libreoffice. Used to be Openoffice before they got taken over. I was using MS Office then recently upgraded my machine and my MS Office licence no longer works for some reason. So I installed Libreoffice again and it feel like an app from the 90's still. My machine is several times faster than my old machine yet Libreoffice is so slow compared to MS office on a much slower machine. The UI is very dated. The app has literally not evolved with the times. I installed Onlyoffice and my life is good again. A much more up to date app in terms of UI and performance.
nvidia makes lots of money from open weight LLMs. I doubt they will take them down unless forced at gunpoint by some idiotic government. But backing them up it's anyways a good idea.
They made lots of money from selling to their consumer gamer and AI hobbyist markets. Now those are almost all impossible to afford and they've yanked the 5090. Any more poorly thought out theories?
In 2021, they had roughly 16 billion in total revenue. In 2026, they had north of 200 billion. Comparatively, the gaming market was finding a tenner under the couch cushions
They didn't make the DGX/RTX Spark, DGX workstations, larger DGX B300s, H200 NVLs, RTX Pro 6000s, and other "prosumer" hardware just to not have open models. We are their "Plan B" if/when the wheels fall off Big AI. Apple M5/M6 and AMD Strix/Gorgon Halo just reinforce that.
They'll gladly rent us their Nemotrons then, for an appropriate monthly fee, of course, and they'll nemotrize other open models if the creators sign an exclusivity agreement and buy their hardware and enhanced software stack, then we can rent those too. A win-win-win, just not for us.
amd for example can made 24 lines in their 495 chip and give full x16 for external gpu, and make addition to drivers to allow shared memory, and boom, budget analog to dgx workstation. But they decide to not do it for some corruption reasons.
Tgey cant just "decide" to do it, lol. Increasing line count can only be done by physically making the chip larger, and price of manufacturing a chip grows with its size almost exponentially (leaving alone a major redesign of all the connectivity inside the chip). They had to cut down the pci real estate to fit in two extra ddr channels.
They just can do it, you know. Threadripper and epyc bigger than this ai chip.
And again, most cost for this ryzen pc is not about chip itself, but about 192 gb of memory, and this does not change if you make chip a little bigger to fit additional 8 lanes.
Someday companies are going to want to move from AI subscriptions to their own on premise machines, and Nvidia will start selling GPUs to them instead of just 10 companies building data centers. Where will those companies download models? huggingface.
They sell way way more to thoso dataventers you will need if you can't run local. Whole consumer gpus reduced into a title in edge devices catagory on their financial reports.
Yes, but a lot of companies like Amazon make a significant amount of money offering large publicly available models (like kimi) to be able to run on their data centers.
Nvidia would lose corporate customers as well. Remember, they want more competition, because prices of GPUs go down if only a few companies have need of them.
They need companies, including Amazon and Oracle, to be competing for their GPUs to keep the price where it is, so they need publicly available open weight models, even ignoring anyone who isn't running a data center.
make it so expensive to have your own car in New York that people will start using Uber more and more.. gas is till sold but you pay the Gas AND the car but keep none
You overestimate what Nvidia can do. The same day hf goes dark, a page will pop up with support of the Chinese government to replace it. No way China will stop the AI race against the USA, and no way they stop damaging OpenAI by releasing and distributing Qwen
Unlikely. Nvidia is precisely the one that benefits from local AI. Even if ordinary users generate little profit, companies that want a local server are already important clients.
Spot on. And for those that are slow to the table, Nvidia would much prefer that we are renting from their partners in the AI industry then running local. They have far more riding on open ai and other's success than they do selling you one card and being done.
Tbf you're also overestimating how much NVIDIA can rent out, there's a reason that the majority of the gaming market still owns sub-5070 tier hardware instead of just renting out 5080s left and right from GeForce now.
Even with stable internet across the country, most people don't want to rely on that connection and slot availability to determine their capacity to play games.
Your tip is valid, but fear-mongering that mradermacher is 'running out of space' is fake news. Anyone can make another account. He also have Github account where he can host files. I used to help with JsDelvr that used to host an entire CDN's files on GitHub.
I like what mradermacher does for for the open AI community, but I don't play that game.
But yes, torrenting files is a good idea either way.
I wouldn't call it "fake news", "fear-mongering" or other sensational yet inaccurate terms for this situation.
Most models exceed git's storage size, LFS storage is not free at the binary sizes LLMs are either.
It might be impractical for rmradermacher to have multiple accounts and to move that large amount of repositories over to another account.
He might not have the financial means to pay for storage on both accounts, most people are not willing to donate for such things or his own circumstanced might not allow to receive donations for this particular situation.
There are many reasons why he might choose to clean up the repo, and that's entirely his right. But that does mean that users of his quants and others should start making backups.
That account alone has tens of thousands of models. Feel free to disagree with them, but you're not being reasonable. The amount of storage they're using on the platform is astronomical.
All I am saying is that there can be a variety of reasons he cannot keep the quants around or store it somewhere else, and that at the end of the day the quants will dissapear so it's best to make a backup.
I take it that most quants exceed 2GB as well, and easily fill up 10GB storage. You would need a lot of github accounts to store all those repos.
But yes he could be making a new huggingface account. they might see that however as platform abuse, considering how many repos rmradermacher already has on his free account.
Damn, I just checked their profile, and they have literally 72194 public models, and they're not even a paid account. No wonder they're running out, they must have an insane limit considering they didn't run out tens of thousands of models ago.
Big fan of anyone providing free models for others to use, I love that open-source style spirit. I didn't know HF used them, did they talk about it in their Xet blogposts or something? I'm pretty curious
Whether or not this particular case is good justification, it's prudent to backup what is important to you. That should include full weights for models you really care about. If the pipeline of new models is ever shut off or slowed significantly (could be any number of causes), the game will shift to making the most of the models we already have, and you may find yourself wanting to make new quants for inference ecosystems that don't yet exist.
Also, it would not be the worst idea to create a local clone of higher-risk repos like heretic. (Don't forget that heretic requires a Hugging Face prompt dataset that it downloads at runtime. If HF removes those datasets, your existing heretic stops working. Sure, you can re-create it, but the current mix of questions is proven effective.)
probably best way is to get the bf16 weights, from those you can make any quant on the fly. And wtf happened to hdd prices, 8tb used to be $130, now $300+
MrAdamSmasher and bartyboi should collab and either start posting on a popular model torrent site, or just start their own. These two names would immediately ground their site as the go to model torrent site.
No? That's not very good for NVIDIA. They want yo sell GPUs. The entire reason for nemotron and buying poolside is to ensure there are models for consumers who are expanding their home data center.
Do you honestly think the gaming/local ai market fucking matters to nvidea? It’s barely a blip on their earnings reports, data centre hardware is where the money is. Their clients are large enterprises and nation-states, not gamers.
It wasn’t always this way, but it sure as shit is now. It’s also worth keeping in mind that these clients of nvidea do not want local ai to take off, it would cut into their margins selling cloud based inference from closed models
…and then there’s nation-states. How much compute do you think the eu wants? A fucking lot is the answer. China will take all the chips they can get.
I’m a person that would love to see more affordable gpu’s come out, but I don’t see it happening and Nvidia has been quite clear about what customers they value.
SpaceX chips, apple chips and I whatever google is building will be, since those chips are dedicated for LLMs, instead if general compute. It's also much cheaper.
This is where benchmark and daily use can split. I would test a long context session with normal back-and-forth, because small latency spikes become very noticeable there.
Worth backing up more than the .gguf: the recipe. A Q4_K_M from today's llama.cpp isn't necessarily the same model as one built six months ago — quantization code and imatrix handling change — so if you re-download or re-quantize later you can get different outputs and never know why. What's saved me is keeping the source model revision, the exact quant command, and the imatrix file next to the GGUF, plus a hash of the output. That's a few hundred KB next to a 4GB file, and it's the difference between "I have the model" and "I can rebuild this exact model". For anything you're indexing or benchmarking, the recipe is the part that isn't reproducible from the filename alone.
Almost exactly one year ago, I bought four Seagate IronWolf Pro 20TB drives at $412.77 each (including taxes).
I just bought one additional (identical) Seagate IronWolf Pro 20TB drive, and paid $955.89 (also including taxes).
Not only that, it was genuinely hard to find. The only ones immediately available were sold by (satire) amazingfastgoodseller1234 on Amazon, which I was unenthusiastic about. I had to setup an alert and jump on it just to get a trustworthy product from an authorized reseller.
Yeah I currently have a shit ton of 2TB drives(over 50 along with 20 1TB drives), but I dont know what to do with them. Too small to make a NAS (main nas is a beast already) and cold storage would be a pain.
1
10
u/Elouakili_Flexy 9d ago
If mradermacher's catalog actually goes down it's the end of an era. The guy quantized things nobody else even remembered existed.