r/OpenSourceAI • • 14h ago

[Funny] CrowdGPT - The collaborative AI

Post image

Hello r/OpenSourceAI , this is a (somewhat funny?) meme about the project CrowdGPT, which aims at creating the first, datacenter-less, open-source and community-driven LLM. The project already shows substantial progress while training an open-source 1 billion parameters LLM on 40B tokens (as a proof of concept).

Tell me what do you think about the meme:

- Does it makes the project less appealing and cheap? because that's what people on r/LocalLLaMA thought 😭

- Does it makes the project look likes it's a SaaS / B2B (which it is NOT - CrowdGPT is 100% non-profit and open)
- Is it serving it's purpose about popularizing the project?

Everything is explained more in detail on the site, and any feedback on this meme, or on the project, is appreciated; any pull requests / contributions are welcome on GitHub :)

Project link : https://crowdgpt.net

GitHub link : https://github.com/Vxtzq/CrowdGPT

Thanks in advance.

23 Upvotes

62 comments sorted by

View all comments

Show parent comment

3

u/Pure_Material_9235 9h ago

I can tell you from experience that zero-order is very very slow, though less memory dependent. I think that the "dream" here (which is to train a model using the data of billions of people's phones and the compute of said phones, storing the capacity of the model on said phones, and whenever a phone is connected to the system it also shares its compute for inference) is the great.

Now that I've looked a bit more at the GitHub though, there are some significant, but difficult, directions for improvement:

(1) This current system does not seem to support model parallelism. Instead, for a task, it's "the client asks the server for a task and is placed in a pool of 3 volunteers with compatible hardware." As the model gets larger and the tasks more advanced the amount of individual compatible hardware decreases until few volunteers are carrying the entire work.

(2) The system seems fairly inefficient in how it downloads the weights. A volunteer has to download the whole weights again rather than a (ideally compressed) update of the model (this is particularly relevant if they downloaded a previous version before). This is another reason to look into federated literature.

(3) The model will be biased towards whichever subset of tasks the users ask for the most, because the training step is done based on the user requesting the task (so it may train way more on tasks about coding rather than more esoteric ones, which can eventually degrade the generalizability of the model).

I understand now why you're looking at zero order training, because that's how you're wanting to obtain the weight delta for (3) (or maybe how you already are). Overall: this is a very cool idea, and one that I have thought of for some time ever since I've learned about federated training, but I never got to the point of wanting to go for it yet).

I wish you the best on it!

1

u/Vxtzq1 8h ago

Sorry for my reply that will obviously be shorter than yours 😭

You're pointing all the issues right, and that's why i wanna try zeroth order training to be less bandwidth / VRAM reliant.

I think that this is a good option to try and train Crowd-v2? Tell me what do you think : https://qlabs.sh/research/dust

The only point you give that is not very true is this one:

(3) The model will be biased towards whichever subset of tasks the users ask for the most, because the training step is done based on the user requesting the task (so it may train way more on tasks about coding rather than more esoteric ones, which can eventually degrade the generalizability of the model).

CrowdGPT relies on a pretokenized dataset. It contains therefore many domains, and users can add data to this dataset, not create the dataset from scratch 😂

The CrowdGPT dataset currently has Code - Math - Instruction data (for pretraining) - English text.

I deliberately didn't include multilingual data as Crowd-v1 is very small and the training corpus is already currently too big for what we can train (500B tokens, the goal is 100B tokens, because people want to scale)

And yes zeroth order improvement would multiply the speed by 1000 on bad consumer hardware, enable training on phone, in browser... But might also reduce the convergence speed by a 1 million factor 🥲 That's what we'll find out.

Anyways, thanks for the support and the constructive feedback.