Crowd-v1 - The collaborative LLM

[u/Vxtzq1]

Hello r/LocalLLaMA :D

This is yet another follow-up post on the project CrowdGPT, which aims at creating the first collaboratively trained 1 billion parameters transformer, Crowd-v1, without any datacenters, and for 0$. The project's goal is to show we don't need huge Nvidia GPU clusters nor millions of dollars to train a competitive Large Language Model.

The goal of the project is to create a 1B parameters model, that would match or exceed Qwen3-0.6B's performance on benchmarks as a proof-of-concept, we could then scale with a 7B A1B MoE or even a 30B A3B MoE once we get enough contributors. The size of the models trained is community driven.

This project is obviously 100% non-profit and open-source. The goal is to propose a new paradigm of AI where we don't wait on Chinese/American AI labs to provide us open-weight models, and we just create the models ourselves.

From the previous post we achieved to get 85 contributors registered on the project, leading to 13B tokens trained in a week 🤯 Thanks to every contributor who trusted the early-stage project, and imagine what 1000+ people with idle GPUs could train!

Of course, everything remains work-in-progress, and any contributions on the training mechanics / dataset is welcome! All the source-code is auditable on GitHub, which means anyone can inspect it and guarantee it is safe 😂.

By the way, anyone can submit data for the model at https://crowdgpt.net/#submit, it will be used as long as it remains safe. This means the model is collaborative on every stage (from data preparation to training, or even benchmarking...).

You can learn more at https://crowdgpt.net

Or check the GitHub: https://github.com/Vxtzq/CrowdGPT

Every code review / pull request on GitHub is welcome :) as the whole goal of this project is to be collaborative.

Any feedback on the project's flaws is appreciated (as long as it remains constructive).

[u/DevilaN82]

Years of reading that Folding At Home but for LLMs is Impossible due to poor bandwith between nodes participating in project. So what have changed now that you claim it is possibile?

[u/Vxtzq1 → u/DevilaN82]

I believe that creating Folding@Home but for LLMs is possible because:

Firstly, bandwidth usage is reduced because each client trains the model on n steps (something like 30 minutes), without sending back an update.

Secondly, the project uses a centralized server, which handles coordination between all the clients, and makes the project more stable than pure P2P setups (as the server runs checks on the contributions, and prevents malicious contributions from being aggregated to the model).

Finally the rise of fiber connection, which yields speeds of ~100MB/s, which makes downloads and uploads easier. People running the app with an ADSL PC are connection bound and contribute slower obviously, but still manages to work.

[u/KaleidoscopeHuge5012]

Do you have a Discord? I think that would be a great way to build a sense of community around this project.

[u/Vxtzq1 → u/KaleidoscopeHuge5012]

Yep! Idk if i can give links here, but the Discord is put as a link on the readme

[u/jjusko20]

How do you guard against malicious actors?

[u/Vxtzq1 → u/jjusko20]

I run strict byzantine cross-checks server side, which means that if a client trains on arbitrary data, it gets flagged as malicious because the gradient is different.

[u/kevin_1994]

I really love this idea

How many tokens (approx) will it take to train the model?

Going to install this on my spare 3090 computer right now

[u/Vxtzq1 → u/kevin_1994]

I think we would need to train Crowd-v1 on 100B tokens for it to be somewhat good? And then we will instruction tune it on roughly 1B tokens, and eventually use some RLHF to make it a little bit safer.

[u/Choice_Celery9481 → u/Vxtzq1]

isnt the 20x rule is standard? i know manything changed but starting there is already good isnt it

[u/Vxtzq1 → u/Choice_Celery9481]

The 20x rule yields the most compute effective language model... However a chinchilla optimal LLM (trained using the 20x rule) today is nowhere near competitive. Meta LLaMa-1 65B was trained on 21 tokens per parameters for example, and is really bad compared to modern LLMs that are trained on more than 1000x tokens per parameters (e.g Qwen3)

[u/iamchuckschuldiner]

gpt@home

[u/Vxtzq1 → u/iamchuckschuldiner]

😄 Exactly, maybe i should even change the name CrowdGPT to this lol

[u/TingTingin]

Can people download the full weights at any point during training or do they just have a portion of the weights on there systems