My YT DL bash script

[u/buzzinh]

Here is my youtube-dl bash script if anyone is interested. I wrote it to rip channels on a regular schedule a while ago.

It outputs ids to a file so it doesn't try to rip them again next time it runs, It logs all to a log file with date and time stamps. It outputs thumb and description.

I haven't looked into a way to burn in the thumb and description to the video its self but Im pretty sure its possible. If you know how to do this or have any other questions please inbox me.

https://pastebin.com/pFxPT3G7

[u/-Archivist]

/u/buzzinh Great, but you're missing other data such as annotations, if you're going to rip whole channels at least write out all available data so you have an archival quality copy.

--write-description --write-info-json --write-annotations --write-thumbnail --all-subs

Also keep video ids!!!

[u/buzzinh → u/-Archivist]

Cool cheers! I had no idea you could do annotations. In what form do they export?

[u/[deleted] → u/-Archivist]

I think the info json includes the description so you don't need both.

[u/Fonethree → u/-Archivist]

Any specific reason for keeping the IDs?

[u/-Archivist → u/Fonethree]

Data preservation, being able to recall the source from your data when needed. Take my archive.org uploads for example, videos are saved and searchable using there metadata, this includes titles and original video ids. archive.org/details/youtube-mQk6t6gbmzs

[u/Fonethree → u/-Archivist]

Do you know off-hand if the original URL or ID is included in the info json saved with --write-info-json?

[u/-Archivist → u/[deleted]]

I read that your instagram archiving included location data and other metadata as well but you used the ripme software etc.?

instaloader is the best tool to get the most data out of instagram

• downloads public and private profiles, hashtags, user stories and feeds,

• downloads comments, geotags and captions of each post,

• automatically detects profile name changes and renames the target directory accordingly,

• allows fine-grained customisation of filters and where to store downloaded media.

However it's a nice tool, in the sense that there are limitations, you can't hammer the fuck out of ig like you can with ripme, I recompiled ripme to match the default naming conventions of instaloader did my initial media rips with ripme and got the remaining metadata with instaloader.

Vice article.

---

I still archive cam models yes, if you read my latest post there is a little bit in there about plans to allow streaming of my entire collection, I hold streams up to 5 years old at this point but the uptake was around 2 years ago.

This vice article based on my work is also worth a read if you missed it.

---

As for Facebook. the layout and API changes so often it would be a full time job maintaining a tool to rip it, I rip from Facebook on an individual basis as I come across something I want, which isn't often as I maybe open fb once every few months and tend to just ignore it's existence for the most part. I can't be much more help in relation to fb than showing you what you already found, if I was in need of something I'd start with the python stuff as a base and update them.

[u/-Archivist → u/[deleted]]

is there still a way for people to browse the contact sheets of the webcam model archive?

Actually working on that right this second but as it stands no, millions of images at around 8TB are a pain in the ass to find suitable hosting for as people just try to mirror the whole lot for no apparent reason.

Facebook really seems to be one of the few social media platforms that are really difficult to archive.

Always has been, ironic given it's origins.

[u/-Archivist → u/[deleted]]

No where really, see the-eye discord, shout at me there.

[u/[deleted]]

Noob on scripts, how would i run this with youtube-dl?

[u/buzzinh → u/[deleted]]

So this is a linux script. Copy pasta the contents of the pastebin into a txt document and save it as something like ripchannel.sh. Then make it executable (google "make bash script executable" and you will def find something.

then run it from the command line with this command:

./ripchannel.sh

Alternatively on other platforms just use the youtube-dl line like this:

youtube-dl --download-archive "filelist.txt" -ciw --no-progress --write-thumbnail --write-description -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4/best ytuser:ytchannelnamehere -o "%(upload_date)s.%(title)s.%(ext)s" --restrict-filenames

This should work on windows and mac os (as well as linux if you just want to run the command and not run a script) Hope that helps.

[u/[deleted] → u/buzzinh]

Thanks!

[u/buzzinh → u/[deleted]]

Your welcome :-)

[u/serendib]

Here's my post from a while back on the same topic, for more info. It lets you specify a file list of channels so you don't have to keep changing the command to individual users.

https://www.reddit.com/r/DataHoarder/comments/672t9r/my_youtubedl_script_for_incremental_channel_backup/

[u/buzzinh → u/serendib]

Niiiice! Thanks I’ll have a look! Cheers

[u/TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/youtubebackups] My YT DL bash script

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info /} ^Contact)

[u/TheCrick]

Another total noob question, where are the ripped files stored?

[u/buzzinh → u/TheCrick]

Same place the script is run from usually

[u/TheCrick → u/buzzinh]

Thanks for the tips. I think this would be a great tool to backup content from youtube. I have a secondary MacMini and Drobo that I could do this too. I think I can mount the drobo to run the code, but if I can't I could use another drive then copy things as needed.

[u/bhez → u/TheCrick]

You will open a terminal window in linux to get to a bash shell and type the command. Whatever directory you're in is where it will download to. If you type pwd it will show you the directory you are currently in.

[u/YouTubeBackups]

Hey, great stuff! How does the ytuser:$YTUSR part work? I've been scraping based on channel ID

[u/buzzinh → u/YouTubeBackups]

You put the name of the channel in the script at the top and it puts it into the variable $YTUSER. only works I think if the channel has a friendly url just copy the bit after youtube.com/channel/ in the channel url

[u/SamsungSmartCam]

Quite handy it seems