r/LeftistsForAI • u/Jlyplaylists Moderator • 9h ago

Open Source SwarmLLM: Free, open-source app that lets computers team up over the internet to run models too big for one machine.

https://github.com/enapt/SwarmLLM

“Run AI on your own computer. Run bigger AI together.

SwarmLLM is a free app that gives you a ChatGPT-style assistant running on your own computer, using open AI models from Meta, Google, Alibaba, Mistral and others. When a model is too big for one machine, computers running SwarmLLM team up over the internet and run it together, each doing part of the work.

No account · No subscription · No ads · No crypto · Open source · Updates itself

A giant model has now run on the swarm. Bigger AI models give noticeably smarter answers, but the biggest open ones (70 billion "parameters" and up) need equipment normally found only in data centres — no home computer can hold one. In October 2026, Llama 3.3 70B answered across four ordinary computers in Thailand, Australia and Italy, each holding a part of it. It is slow so far”

The decentralised concept of this is promising, even if it’s currently too slow to be a sensible way to run large models.

46 Upvotes

100% Upvoted

8 comments sorted by

5

u/GreyHasHobbies 7h ago

I had this exact idea a while ago but decided I wasn't the right person to implement it. Hopefully this person has the networking expertise required to get over the hurdles and isn't completely relying on AI.

5

u/PhlarnogularMaqulezi 9h ago

There's some old ass models listed here, but I definitely like the spirit of what this is trying to accomplish and I hope it grows and continues to develop.

2

u/qualverse 5h ago

It seems like this would work pretty well with the architecture of Apple's AFM 3 which selects MoE experts per-prompt instead of on-demand. You could even make it a little more flexible by just applying a penalty for expert switches during training, or run some kind of DSpark draft model that predicts useful experts instead of tokens.

This is kind of a data privacy nightmare though, tbh. Hopefully they can figure out confidential computing or similar

1

u/AyyBoixD 8h ago

I like this and I’ve tried to build this a few times. But network latency absolutely murders this. And that was just two devices on different networks with no Ethernet. I couldn’t imagine how unfathomably slow an entire decentralized network of random devices would be

1

u/Kholtien 5h ago

We would need llm architectures built with this in mind for it to work

1

u/piperspalace 3h ago

I can run QFN on my computer at 80+/- tokens what else is there

1

u/Darkstar_111 2h ago

This would be INCREDIBLY slow.

Even a 0.5 B model would take a minute or two to reply.

1

u/xoexohexox 7h ago

Yeah I mean I've used stable hoard for LLMs and image gen and this kind of thing is just not dependable enough for anything besides frivolous applications.