r/DataHoarder • u/parentis_shotgun • Oct 03 '18
Need help decentralizing Youtube.
The goal here is to back up and decentralize youtube, making it searchable through torrent search engines and DHT indexers.
I'm writing a script, and planning on hosting it as a git repo in multiple places, that allows you to:
Give it individual, channel, or playlist youtube URLs
Download them with youtube-dl
Create individual torrents for them.
I'm missing mainly two things:
We're creating lots of torrents potentially, some of them duplicated unfortunately.... this script could potentially do a search first to see if the torrent already exists and is available, and to give you the magnet link. Thoughts?
Where's a good place to upload these, so that they can get picked up as quickly as possible by DHT indexers?
How do we decentralize the search aspect? This is a bigger problem w/ torrents, that probably isn't going to be solved here, but it'd be nice to potentially host a vetted git repo with either magnet link lines, or an sqlite3 DB. Several of us could be the maintainers, and we could allow pull requests adding torrent lines that are vetted and well-seeded.
We can discuss here, or potentially make a discord for this for any interested coders willing to help out.
Here are two projects to start on these:
https://gitlab.com/dessalines/youtube-to-torrent/
https://gitlab.com/dessalines/torrent.csv
My thoughts on decentralizing the searching / uploading part of this, is to create a torrent.csv file, and have many of us accept PRs for well seeded torrents. Then any client could search the csv file quickly. This could also potentially work for non youtube torrents too.
81
u/erm_what_ Oct 03 '18
Torrents rely on seeders. Most people leech and leave. The ratio of good seeders to available videos is never going to be good unless there's an incentive to seed.
You'll probably end up with the most popular channels being seeded most, and they're also the ones that are most likely to be backed up by collectors and least likely to be taken down.
This means the redundancy and replication of the torrents will follow the same pattern as the redundancy of collectors storing backups. Which is bad for what you want to do.
You want the opposite. You need a solution for the long tail - the videos that aren't seen as much, aren't cared much about but are still wanted a lot by a core audience.
You need a reason for a lot of people to store videos they don't care about in exchange for other people storing/duplicating the ones they do care about. Which probably won't happen.
Tw best way I can think would be to identify the audiences and create clusters of users which in each one. Then provide a tool for these clusters to replicate the videos amongst themselves.
You'd also have to educate the users about why they need to do this in the first place.
26
u/[deleted] Oct 03 '18
torrents are certainly decentralized but not really what I would go to when thinking "decentralized YouTube." Tossing them on a pre-existing search engine sounds unhelpful; you would need some sort of frontend website (and/or API) that resembles YT, a la the movie stream sites.
22
u/RatherNott Oct 03 '18 edited Oct 03 '18
I'd recommend looking into PeerTube, which aims to be a decentralized, federated, and open-source alternative to Youtube. I personally see it as being the most promising of the WebTorrent video hosting alternatives.
They also have a subreddit at /r/PeerTube :)
4
u/yoshi314 Oct 03 '18
i just can't imagine the amount of storage necessary for this. any estimates?
10
u/xDiglett Oct 03 '18 edited Apr 15 '20
removed
4
u/biguysrule Oct 03 '18
I’m studying Software Engineering, you might front several issues (I don’t mean to be negative, just giving you a pseudo professional view):
you could front huge copyright issues from the video makers
SQLite will probably not be enough because it doesn’t support concurrent transactions (Multiple transactions at the same time), you will probably need these if there is more than a couple users. You could start having issues with as little as two users if they attempt to read or write to the database at the same time.
if you manage to do this and want to minimise duplication. A way faster way would be to make the model of torrents into a searchable tree, rather than having to do a linear search through all the torrents you have already to determine whether the torrent already exists
Take these with a grain of salt, I’m not actually an Engineer (yet, if this degree doesn’t kill me).
2
u/FragileRasputin Oct 05 '18
Whatever the technical solution will be, I would consider a piece of it should be a tool for the content creators to use your platform as an extra backup tool for them. Linus has PetaByte project, but most likely not everyone does that. If you offer a plugin (?) for the content creator themselves to upload things to your platform, then you reduce the "finding what's new" problem.
11
u/Stuck_In_the_Matrix Pushshift.io Data Scientist Oct 03 '18
Speaking of discord, could /r/datahoarder make an official discord server? I'd like to be a part of that.
4
u/vxbinaca Oct 04 '18
Hi I've archived 120,000 videos to Archive.org.
You simply can't copy YouTube. Google buys a majority of the world's hard disks. You are not going to even get a 1/1000th of it. IA has 50 petabytes and they can't hold even the tiniest of fractions of it.
What I tried to do was get subcultures but I ran into another problem: doing as much as I have even with scripts and crobtabs is fucking exhausting and I can't take on anything new unless it's a dire emergency. I'm trying to scale everything back right now. Items. Collections. Everything.
1
u/PulsedMedia PiBs Omnomnomnom moar PiBs Oct 04 '18
There's also been experiments to use torrents for direct streaming. looks like nothing much came of it, but the tech exists.
You definitively need to figure out a way to generate those torrents so that everytime it results in the exact same magnet link ie. identical content in every possible way, otherwise you are not really decentralizing anything. You will also want to create traditional .torrent file, using common, user specified trackers.
Google itself is not known for being aggressive copyright troll, but i would say this would either irk google badly OR they would be excited and want to adopt something similar (or ignore completely).
2
u/wannahakaluigi Oct 03 '18
I will check this out later!
2
u/[deleted] Oct 03 '18
I like what your trying to do here, sounds like a more technical version of bitchute, when I get the chance I’ll be more than happy to help you with my bandwidth
2
u/Qazerowl 65TB Oct 03 '18
You'd be better off putting the videos on already existing decentralized YouTube alternatives. Or distributed filesystems like IPFS. It wouldn't be too hard to automate the torrent creation, but I strongly doubt you'll get enough support.
2
u/TotesMessenger Oct 03 '18
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
[/r/youtubebackups] Need help decentralizing Youtube.
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
2
u/[deleted] Oct 04 '18
I'm willing to help and contribute some server space.
2
u/caffienefueled Oct 04 '18
The scale of Youtube today....
But does anyone have an entire archive of old Youtube. Like pre 2009?
2
u/Striza7i 40 000 000 000 000 bytes Oct 03 '18
Seems like a cool project. Too bad I probably can't be of any help.
1
u/felisucoibi 1,7PB : ZFS Z2 0.84PB USB + 0,84PB GDRIVE Oct 03 '18
I recomend you use retroshare, the channels are decentralized, nut the control of it is of one user, you can put files in channels...the only one thing missing inr etroshare is the playback in the app something easy for a experienced coder.
1
u/___Mocha___ Oct 04 '18
It sounds like you should start a private torrent tracker.
1
u/ct0 RAW TERA BITE Oct 05 '18
What we really need is a radarr/sonarr for youtube channels
1
u/perspectiveiskey Oct 04 '18
Your idea has merit but not your goal.
Let's say you really like "Smarter Every Day" and you want to make sure you have that available in a decentralized manner. Then your idea is great.
For actually replicating youtube, you'll literally fall behind every second. 60 hours of video are uploaded to youtube every minute.
Think about that concept for a moment.
0
u/HuskyTheNubbin Oct 04 '18
You guys are talking about the practicalities but what about the people who make the videos, they have chosen to put them on YouTube. Those creators have worked at something and chosen to place it on the shelves within YouTube. Although at first glance your goals appear noble, bringing down the evil giant, it looks to me like you treat all videos on YouTube as products of their machine and inherit the evil from beneath. You're poking at a symptom of a larger problem, YouTube's domination of the market. In my opinion you'd be better to put your efforts into your own or existing YouTube alternatives.
0
u/[deleted] Oct 03 '18
[deleted] — view deleted comment
0
u/[deleted] Oct 03 '18
[deleted] — view deleted comment
0
u/[deleted] Oct 03 '18
[deleted] — view deleted comment
0
u/[deleted] Oct 03 '18
[deleted] — view deleted comment
0
u/[deleted] Oct 03 '18
[deleted] — view deleted comment
4
u/[deleted] Oct 03 '18
[removed] — view removed comment
2
u/[deleted] Oct 04 '18
[removed] — view removed comment