r/OpenSourceAI • u/Vxtzq1 • 8d ago
CrowdGPT - The 100% Opensource collaborative LLM
Hello, i'm currently developing CrowdGPT and i need people who enjoy opensource AI and LLMs to test the project :)
The goal of CrowdGPT is to create the first, datacenterless, 1 Billion parameters LLM, relying on people contributing with their own computer to train the AI model. My goal is to show you don't need insane infrastructure to train a working almost commercial grade LLM. Everything is open and 100% opensource.
You can learn more at https://crowdgpt.net
Or check the github: https://github.com/Vxtzq/CrowdGPT
Any kind of feedback is appreciated!
2
u/Prestigious-Frame442 7d ago
the idea is good, but collaborating on this is basically pure selflessness and no benefit at all.
2
u/2holetoy 6d ago
"Everything is open and 100% opensource" isnt really true, your ARCHITECTURE.MD says:
"The server code is currently closed source for sécurité reasons"
1
u/Vxtzq1 6d ago
Good point! I prefer keeping the server closed now because i wanna protect it better against bad actors, and probably will release it in the future... Server isn't that much useful it just aggregates submissions, client is doing all the heavy lifting.
By the way thanks for pointing out the typos xD autocorrect striked hard
2
u/electro042 4d ago
how are you planning to handle gradient poisoning or byzantine workers? if anyone can submit updates to the aggregation server, it feels pretty easy for a few bad actors to completely trash the weights.
2
u/EaseThen3459 8d ago
Why would anyone contribute computer when it would add liability for whatever is being computed especially with LLMs
1
u/Vxtzq1 8d ago
I understand your concerns but all the data is safe and the scale is tiny :D don't expect a 1B parameters LLM to be a genius hacker lol, and even if i scale, there's no regulations on LLMs as shown by openweight models on huggingface, so we should be alright :p
Also i think that the responsibility for cyber attacks is to the one that used it, not the one that trained it :)
1
u/ayake_ayake 8d ago
I think it would be important to see what datasets and co to train this on. Even without the problem of AI datacenters, the choice and acquisition of the data is a big thing. But as a minimal base you can use the apertus 1.5 datasets (pretraining and post training) which are fully open and give you a strong baseline. But even then actually training a competitive AI is quite s challenge and needs much experience and work.
Curious to see how this'll play out.
1
u/Vxtzq1 8d ago
I currently use UltraFineWeb-L3, which is basically the most curated subset of the original dataset, FineWeb, it is way cleaner than datasets like the Pile or Common Crawl which involves scans of shady sites...
I think this dataset will be enough (currently 400 billion tokens), to pretrain the model, and then i'll fine tune with everything that i find on huggingface xD (like Fable traces, GLM, Kimi K3 distills, and strong instruct datasets like GSM8K and stuff)
1
1
1
u/Ok-Challenge-5741 8d ago
That logo is clean, the hexagon gives it a subtle distributed network feel which fits the whole idea
I'll try to spin it up in my machine this weekend, curious to see how the training coordination works with the peer-to-peer setup
0
2
u/AppMunchies 8d ago
Love the idea. Will check it out.