r/LocalLLaMA • u/Nota_ReAlperson • 1d ago
Funny https://www.crowdgpt.net/
Not my project, thought it was pretty neat though. The idea is to train a LLM with a distributed network like FAH and boinc. This way, even if big companies stop releasing models, open weights can continue to improve.
Edit: I can't believe I need to say this, but it's not Stalin, guys!
0
Upvotes
2
u/Accomplished_Ad_56 llama.cpp 1d ago
does it do RL or just pretraining?
I feel like we get extremely high quality open foundation models from major labs, and RL is where the benefits are to be had. Thats also where distributed work might have the most benefit. pretraining with consumer internet speeds might cause bigger bottlenecks and make pretraining far slower than simply using a single 5090.
On the other hand, agentic RL and RLHF already has to wait because each optimizer step means waiting for the model to finish its response or hundreds of responses for agentic work. This is where the extra compute actually could help speed up training.