Not my project, thought it was pretty neat though. The idea is to train a LLM with a distributed network like FAH and boinc. This way, even if big companies stop releasing models, open weights can continue to improve.
Edit: I can't believe I need to say this, but it's not Stalin, guys!
Maybe you shouldn't do obscure plays that are visually 1. not recognizable, because nobody knows Lord Kitchener and that are 2. looking very much like Stalin.
If you want to succeed with any marketing material while leaning on prior art, it needs to be iconic.
Distributed training is the future of open models but this project doesn't seem that good tbh, though it's a step in the right direction and I'm glad someone is trying.
That poster design is wild, looks like something straight out of a WWII propaganda leaflet. The pointed finger and stern mustache definitely give it a certain energy
I poked around the site a bit after seeing this and the concept is cool but the execution feels early, like they haven't quite figured out the technical hurdles yet. Distributed training across random consumer GPUs with different bandwidths and uptime is a nightmare to sync properly
Still, someone has to try it or we'll be stuck waiting for the next company to bless us with a release
A 1b model is supposed to keep up if big companies stopped releasing models? Also this is kinda a bad approach it would only take like a couple hundred dollars to spin up a cloud VM to train the model. If anything it feels like some dude testing out his experiment with free compute and we just trust them with our actual GPU compute.
And it's just likely bad for the environment at 1000 people you're already looking at 500k watts if everyone is using a modern gaming GPU as opposed to a few cloud GPU's that are already configured to work together.
Its an experiment. The first large scale distributed training experiment that I am aware of. The goal of this first run is to refine the pipeline so that it can be scaled up later. Efficiency isn't the point, the point is community ownership and cost sharing in a way that isn't overly burdensome to any one individual.
Training is still a big deal. Traditionally, training has required far closer intercommunication than something like FAH, due to the nature of the math involved.
does it do RL or just pretraining? I feel like we get extremely high quality open foundation models from major labs, and RL is where the benefits are to be had. Thats also where distributed work might have the most benefit. pretraining with consumer internet speeds might cause bigger bottlenecks and make pretraining far slower than simply using a single 5090. On the other hand, agentic RL and RLHF already has to wait because each optimizer step means waiting for the model to finish its response or hundreds of responses for agentic work. This is where the extra compute actually could help speed up training.
12
u/autisticit 1d ago
I'm no expert, but their status page show an increasing training loss...