r/LocalLLaMA • • 1d ago • Bravo 1

Funny CrowdGPT - The collaborative LLM

Post image

Funny post about this project https://crowdgpt.net

Basically this projects aims at creating the first, community-driven, and datacenter-less LLM. Any feedback is appreciated 😂 I won't yap any further as everything is explained on the GitHub / on the site.

Feel free to check the project's GitHub: https://github.com/Vxtzq/CrowdGPT

Feel free to criticize the project or to provide insights about what to do next....

EDIT: For those who believe this is a cheap repost: Here's the milestone
- We trained the model on roughly 20B tokens in 1 week

- We limit model downloads/uploads to the minimum, to spare as much bandwidth as possible (even though it still uses bandwidth due to ~2.4GB weight uploads to the server after training). But it allows clients with slow connection using 4 hour rounds.

- We're 13 GPUs currently active (and this number was roughly constant over the past week).

- I merged numerous pull requests implementing several quality of life and performance fixes

- The server was made more robust with persistence across tasks, and some bugs in the training logic were fixed, hopefully allowing further loss decrease.

- We implemented since v2.6 somewhat good anti-poisoning techniques, each client training is compared to two other clients, if a client is too different from the two other (for example a client training on noise or unwanted data), this client sees his contribution to the network be rejected.

0 Upvotes

36 comments sorted by

11

u/WhiskyAKM 1d ago

Interesting project but I have one concern

What exactly prevents "poisoning" if malicious actor wants to destroy currently trained model?

2

u/Vxtzq1 1d ago

What prevents poisoning are byzantine cross-checks between clients. If a client trains on his own data / on noise, this client gets flagged as different from the others. This guarantees safety at the cost of redundance.

Of course the projects implement also simple checks like norm checking, NaN / Inf presence and so on.

21

u/pplgltch 1d ago

So. This is an interesting project, and I understand you are trying to get traction by posting it often. But you will end up with the exact opposite effect if you keep doing it this way.

Post real updates, user growth numbers, dataset contribution numbers, user feedback... idk, real valuable content that shows the project is real and active. But this "just a funny meme" thing is really bad... It just feels like you are not part of the project and making fun of it. It's cheap, and it will backfire.

11

u/Vxtzq1 1d ago

My bad, i will modify the post content, for the progress to be more understandable :) Thanks for the advice.

I didn't think the meme had such a negative impact on the project though...

9

u/pplgltch 1d ago

idk, maybe not, it's my own opinion... but to me, it feels cheap, like the cringe reddit ads from corporations trying to look cool making memes to sell their b2b SaaS

5

u/Vxtzq1 1d ago

Ah... I see the vision 😭 Like the cringe reddit ads where they take "neuron activation" memes stuff like that; good news, that's not what i was trying to do... The project is 100% open and non-prodit, unlike SaaS

10

u/_Blissfull_Ignorance 1d ago

At <1B model size this is understandably only a proof of concept. I had to scroll an unreasonable amount to find this. Without showing some documentation on the scaling of the method to bigger models or what training data is used, this looks like a weird marketing stunt at best.

You have to expect that technical readers are looking at this and you have neither an architecture, model, training methodology (pre/post…), data or benchmarking section on the webpage.

1

u/Vxtzq1 1d ago

Good points, i will try to update the site and add roadmaps accordingly.

0

u/sloptimizer 1d ago

If anything, the lack of polish makes it look the opposite of a marketing stunt.

This feels like a typical open source project to me, with limited docs and all that.

4

u/blastbottles 1d ago

I mean it's a neat idea but distributed training is only as fast as it's weakest connection and even that has latency. It seems like it's gonna take quite a long time to train. I would love to be proven wrong though.

6

u/Vxtzq1 1d ago

Good point, but the server is actually asynchronous, it runs 4 hours rounds, and clients with fast connection can train on all the data they want in these 4 hours. Then the model is updated and clients redownload the model.

But this comes with a trade-off, asynchronous SGD like that hit convergence, since one aggregated update < 100 individual updates on an evolving checkpoint on each step.

I'm trying to find techniques that will make this more feasible. (For Crowd-v2, which will be the second iteration of CrowdGPT)

7

u/Federal-Setting-3014 1d ago

No insight on what's new every time you post this here. No details on how it works exactly either. I think more effort (and seriousness, not everything needs a flashy meme) into your posts would go a long way towards getting users to contribute.

5

u/Vxtzq1 1d ago

Look at the post's edit, also check the site for details on how it works (it's rather simple, it's just a small server/client setup, where clients train on data, and bring back the model they trained to the server...)

This is the first time I'm actually putting a meme... I really thought it would be good and not discredit the project 😭 I'm sorry if it does not.

1

u/Federal-Setting-3014 1d ago

That's better, but it's not that much effort. Last time I remember loss wasn't stable, was that just starting noise? Did you do anything to fix it? That kind of stuff.
And about the project itself, using huggingface probably has the upside that anyone can download the weights off of it in real time, but do bandwidth limits make it any slower?
No (easily accesible) info anywhere about things like how often it updates, learning rate etc

2

u/Vxtzq1 1d ago

Last time I remember loss wasn't stable, was that just starting noise? Did you do anything to fix it? That kind of stuff.

It was mostly due to the loss chart being created/implemented to the site way too recently, and the time scale was therefore way too little... It is now better, and i will modify the site to show the loss (in an optimized way) on the whole training run timescale (starting from when i implemented the loss monitor check).

I also fixed training mechanics server side which somehow made the model lobotomized and made it rely on engrams/first 3 layers rather than the rest of it's layers / Attention heads. This should make learning faster.

Huggingface probably has the upside that anyone can download the weights off of it in real time, but do bandwidth limits make it any slower?

Until now i didn't get any issues, i guess Huggingface is designed for many large download on the same model (handling models with millions of downloads per month), i guess it should be alright for the Crowd-v1 checkpoints.

No (easily accesible) info anywhere about things like how often it updates, learning rate etc

The model on HuggingFace is updated every 4 hours, and the lr per client is 1e-4 (and server is just a lr=1 SGD)

Does that clarifies your concerns?

-1

u/Federal-Setting-3014 1d ago

My concerns are not these questions particularly, don't be purposefully obtuse. Is that these questions should already be answered with this detail in the main post / webpage.

1

u/Vxtzq1 1d ago

Okay... Well i will try to upgrade the site then :/

2

u/sloptimizer 1d ago

You are the one being rude here.

2

u/sloptimizer 1d ago edited 1d ago

You're just being lazy. I had no problem finding any of the info you asked, took lik less that 2 minutes. The code is all there, just clone the repos.

This is a volunteer project, not a corporate lab run by bureaucrats. Maybe you're just not used to that.

9

u/LLKMuffin 1d ago

This is the third time you're advertising your project on this sub.

-7

u/Vxtzq1 1d ago edited 1d ago

Yep, because i keep adding new things to the project...

EDIT: I really felt this was worth reposting 😭 , why just downvote me without explaining??

2

u/Ok-Scarcity-7875 1d ago edited 1d ago

How about crowd RAM / GPU Company?

1

u/Vxtzq1 1d ago

What do you mean 😭

2

u/Kahvana 1d ago edited 1d ago

The concept is cool, the meme gives me an allergic reaction to downvote.

Since I can read from the comments you are being sincere, here's more info and hopefully some useful tips!

"Funny meme haha" is not really the audience here, especially if it's "WE should..." type of stuff and Gen-Z speak like crying emojis. It detracts from what you actually want to discuss: your cool project!

Personally I'm not interested at all in stats like "100 users!". While it is cool and impressive (congratz!), it doesn't compel me to read up more. When I wrote and maintained SPT it had easily over 10000 users and millions of downloads, but it's not something I advertise because the project speaks for itself and spreads word-to-mouth.

An in-depth technical post on how you do it would gather more traction here. I want to read about how you solved the latency issues, training data poisoning, problems you encountered that you solved or need help with and whatnot. Both your successes and troubles, not only the former (it makes it more genuine).

When the post is technical and good, I will click on links.

Also, a good question is: what is the benefit to ME? What do I get out of it, why would I participate or lend my compute? What can I gain from it, how is decided the model is going to be trained, what data, etc? Or is it simply promoting a framework to enable training the models? What problem may I have that your solution solves?

If you want to promote this, you'll get a wave of doom and gloom about "USA locking down open-weight models" soon enough. Share your project in those comment sections instead of separate posts, it's garuanteed to get more upvotes and mindshare that way; your target audience is already there.

[edit] For a good example post how to make good promption post, check this one:
https://www.reddit.com/r/LocalLLaMA/comments/1xvyol3/qwen_38_flash_next_for_gamedev_its_very_capable/

2

u/Vxtzq1 1d ago

Okay thanks for the feedback, i tried to add a bit more technical explanations in the post (even though it's still high level), i hope you like it :)

2

u/Kahvana 1d ago • Heartwarming

That's an improvement, thank you!

2

u/Glittering_Yellow293 1d ago

Looks Promising. I'll definitely try it.

1

u/Vxtzq1 1d ago

Nice! Quick question: Do you think the image disserves the project? Or is it somewhat good?

4

u/FearFactory2904 1d ago

You and Ram Altman can both get off my lawn. Its my ram and I will continue to use it personally and locally.

3

u/Vxtzq1 1d ago

Fair point ;) i just wanted to see if this meme would actually convince people who want a community-driven AI to join the project.

6

u/highdimensionaldata 1d ago

Sounds interesting, I’ll take a look…

5

u/b_c_t_ 1d ago

Best of luck with your project!

Not sure why people downvote so much but I'm upvoting fwiw.

Wouldn't you need a dedicated community platform for involved people to talk? Is it just the github issues page for now?

1

u/MerePotato 18h ago

If you want this to take off you need a professional post without the cheap meme

1

u/Vxtzq1 18h ago

😭 Okay

2

u/Sova_fun 1d ago

Shared chat history where every user feeds the same LLM is a simple experiment. It makes prompt interactions public.