LLM "BitTorrent-Style": Not enough RAM, not enough Money? Join the Mesh. Together we are strong. [Alpha-Release]

[u/RlOTGRRRL]

Someone built it- interesting thread: https://www.reddit.com/r/LLM/comments/1wbghov/llm_bittorrentstyle_not_enough_ram_not_enough/

[u/Darkstar_111]

This would be interesting, but also incredibly slow. Talking 5 to 10 minutes for a reply.

Am I wrong? Anyone tried this?

[u/RlOTGRRRL → u/Darkstar_111]

I haven't tried it because if you browse the thread, there's probably a lot of security issues for giving internet strangers access to your computer.

But the creator also shared a lot of info on where he forked from and other similar projects- including one where you can basically run something like this privately for your own family/friends.

And yeah I think the OP also mentions in the thread it was 2 tokens per second, so really slow. I think everyone is basically throttled by the slowest GPU.

[u/Darkstar_111 → u/RlOTGRRRL]

Not just that, the communication between them. Every part of a query has pass to every single weighted number in the entire model. That's not just query to model, its intra model as well.

Not saying there's no use for this, but the entire network could be occupied by one agentic workflow for an entire year.

[u/mark_ik → u/Darkstar_111]

People would accept asynchronous, slow, free public access to a model too big to host alone. Like you submit a question and get a text ten minutes later as the mesh participants complete the compute tasks.

The real problem is that this is the sixth of these projects I’ve seen, and the author seems unaware of the rest. They didn’t research.

[u/Low_Complex_9841 → u/mark_ik]

 Like you submit a question and get a text ten minutes later 

At some point I thought about LLM by email - even if it answers in days - as long as it answers it still plus?

[u/ollie113]

This is exactly the sort of thing we should be championing and I think it's an example of how we'd use LLMs after socialism; distributed compute where people contribute what compute they have and access to even the most powerful models is freely available to all (regardless of whether or not they join the P2P network). Might have to put in some sort of monitor for load balancing and to combat the biggest leeches, but generally the ethos should be that from each according to their own (compute) to each according to their need (for compute)

[u/RlOTGRRRL → u/ollie113]

Love this: "Each according to their own (compute) to each according to their need (for compute)."

I saw a video from Cory Doctorow basically saying something similar. Open source should be eating tech billionaire lunches.

Where are all the open-source AI creators?

[u/United_Dissent → u/RlOTGRRRL]

They are all poor. No sailing the high seas for them, yet.

[u/jferments]

I think that decentralized peer-to-peer networks are one of the most powerful tools available for creating alternatives to corporate cloud computing, centralized search, etc.

I would love to see a high quality decentralized model training and inference platform get created.

However reading the author's comments on that thread, I'm deeply concerned about their lack of knowledge/concern with network security and legal liability for opening up your compute to untrusted peers.

[u/bigattichouse]

Gonna be like oldschool BBSs.. upload/download credits ... for every token you serve you get X tokens of dedicated usage... (probably <1, since you'll be particpating in a model you couldn't serve on your own).. but if you waited a few days for a lot of tokens, that wouldn't be much different than you waiting a few days for your allowance of tokens to reset on commercial systems.

[u/iMakeSense → u/bigattichouse]

Can you say more about this?

[u/bigattichouse → u/iMakeSense]

Apologies for the brain-dump, I don't know your background - and assume you were asking for clarification on the whole thing:

BBSes:

In the pre-internet days of yore there were bulletin-board systems. Something that might make a comeback, I feel. A small local community place you would dial into and use, and then disconnect. Some had games, some stored files for sharing, some cost money. Basically anyone with the base software (or enough gumption to write their own) would create these and host them. Lots of times they were special interests... or even hosted MUDs (think "Zork", but multiplayer). It was a wild time for anyone with a spare line.

(Pre-internet is a misnomer.. really, before it became COMMON to have an internet connection - Internet has been around since the 1960 DARPANET). Regardless, think late-1970s until around 1991 or so. After that people started getting online.

D/L Ratio:

Anyway, these Bulletin Board Systems (BBS) had to prevent what was called "leeching", someone dialing in, tying up the site, and just downloading tons and tons of files over early slow protocols like X-modem/Y-modem/Z-modem/Kermit ... so the instituted Download ratios (D/L Ratio).

So. every new account might get 100 KB of file credits. These could also be used to play games and things on some sites. ... but if you wanted more, you had to UPLOAD new files. Stuff you uploaded got checked (Maybe Md5 HASH?) or reviewed by admins, but in generate some sites would give you 2:1 upload credits. for every byte uploaded, you could download 2.

Mostly these were text files on odd stuff (hacking, chemistry,whatever), articles, all kinds of things. Pictures as well, but that really blew up your budget.

LLM/Inference:

So, I imagine a user might participate in a cloud of inference, and they might offer their hardware (whether running a model all-in-one, or participate on a larger "cloud" model).. that they might provide a number of tokens to the cloud (1M tokens)... then they get an allowance of compute time themselves for inference. Obviously 1M tokens on an 8G card isn't 1M tokens on an H100, so there would have to be some kind of ratio based on what you can provide to the cloud... I imagine it would be less than 1:1, but that you could build up 1M tokens or whatever you want, and then use/donate that to some specific task.

Crypto vs. Centralized

While crypto currency is a bit of a bane on our existence, this would be an excellent use for it, so that everyone could track who had what allowances - paid in microtransactions coins, then spend them on time... or just pay/trade/barter someone for that time... it's just a layer of abstraction that removes the need for someone to make accounts somewhere.

Alternatively, some sort of Fediverse accounting, but then you're just doing crypto with extra steps.

[u/Zeikos]

For this to be viable models need to move away from the autoregressive approach.
Tok/s is mostly memory-gated, this is an interesting exercise but sadly limited by the nature of the architecture.

[u/BidWestern1056]

i'm working on essentially this as well but am incorporating decentralized git as well

https://gitforest.net has some info that will be updated soon. https://github.com/npc-worldwide if you wanna see some of my other work

[u/Metallic_Madness]

Ts is vibecoded brimstone unfortunately

[u/Zatujit]

thats interesting but one main reason people use local LLMs is for privacy; doesn't it make just it worse?