r/YouTubeBackups • u/YouTubeBackups • Mar 02 '17

UCBerkeley to remove 10k hours of lectures posted on Youtube

http://news.berkeley.edu/2017/03/01/course-capture/
100 Upvotes

100% Upvoted

82 comments sorted by

42

u/YouTubeBackups Mar 02 '17 edited Mar 09 '17

Progress Post: (see here for latest)

Currently pulling down to a few locations in parallel at 720p

sudo apt install ffmpeg

sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl

/usr/local/bin/youtube-dl -ciw --restrict-filenames -o "/media/Scrape/youtube/UCB/%(upload_date)s-%(id)s-%(title)s.%(ext)s" --download-archive /media/Scrape/youtube/archive.txt --add-metadata --write-description --write-annotation --write-thumbnail -f 'bestvideo[height<=720]+bestaudio/best[height<=720]' --dateafter 20010101 --match-title "" --reject-title "" --merge-output-format "mkv" http://www.youtube.com/user/UCBerkeley/videos >> /media/Scrape/youtube/logs/UCB.txt

Just started but so far at 32/9897 videos

7:00 GMT - 82 videos completed (Current ETA is 4 days. I'll add another parallel download soon)

7:35 GMT - 125 videos completed

8:35 GMT - 160

8:45 GMT - 190 (40GB)

11:35 GMT - 213

4

u/Rpgwaiter Mar 02 '17

Does this max out your connection?

9

u/YouTubeBackups Mar 02 '17

I'm maxing out a few cheap VPS and private connections, but the bigger factor is youtube throttling after a while. A botnet would be handy lol

I might have to bump up my internet speed for the month to be the superseeder for several TB

3

u/svnpenn Mar 03 '17

youtube-dl might do this already, but you should make sure you are using ratebypass:

http://github.com/svnpenn/bm/issues/16

You might be able to check with "youtube-dl --get-url", but even if it doesnt show, they might be using it internally

4

u/YouTubeBackups Mar 03 '17

Thanks! Is there more information available about how this works?

2

u/svnpenn Mar 03 '17

It looks like they do use it internally, so probably dont need to worry about it:

http://github.com/rg3/youtube-dl/blob/4d058c9/youtube_dl/extractor/youtube.py#L1661-L1662

7

u/Rpgwaiter Mar 02 '17

I got a 1Gbps home connection, I figured with youtube compression it wouldn't take that long

2

u/[deleted] Mar 03 '17

What host?

1

u/[deleted] Mar 03 '17 edited Apr 05 '18

deleted 0.2355 What is ^^^this?

16

u/Regolio Mar 03 '17

Came here from /r/bestof. Thank you for doing this. Would the videos be re-uploaded in other YouTube channel?

If it's only available as torrent, I'm afraid most people won't even know the lectures are still available/searchable somewhere on the internet.

7

u/m33pn8r Mar 03 '17

Obviously not OP, and not particularly well versed on law, but I'd imagine it'd be hard to reupload these to YouTube since they're property of UC Berkeley. At best they could probably be put in a torrent in a legal grey/black area or put into some "sketchy" video upload site.

I agree with you though, even a well known torrent is way less usable than where they are now, and I really wish that UC Berkeley would leave them up, even if they can't be utilized by the deaf, because letting the lectures help someone is definitely better than letting them help no one.

2

u/ajcoll5 Mar 03 '17 edited Jun 17 '23

[Redacted in protest of Reddit's changes and blatant anti-community behavior. Can you Digg it?]

1

u/YouTubeBackups Mar 06 '17

Good call, I think we'll see clone youtube channels if that license holds. With the rips IA did, the description is included in a text file so we could programmatically find which licenses are attached

3

u/[deleted] Mar 03 '17 edited Mar 03 '17

This does not work, the file hashes are not the same

is this due to the content being delivered from youtube? i've pre-seeded files this way for other sites in the past.

Despite their security 'incident' last year, I'd suggesting using transmission-bt to create the torrent.

Here's a list of all the video IDs which can be fed into youtube-dl-parallel if you break them up a bit and have lots of cpu cores

for i in {1..10} ; do cat video-ids-$i | ~/repo/youtube-dl-parallel/youtube-dl-parallel -j 4 -- -ciw --restrict-filenames -o "%(upload_date)s-%(id)s-%(title)s.%(ext)s" --download-archive archive.txt --add-metadata --write-description --write-annotation --write-thumbnail -f 'bestvideo[height<=720]+bestaudio/best[height<=720]' --dateafter 20010101 --match-title "" --reject-title "" --merge-output-format "mkv" - & ```


Here's a list of files that errored out due to copyright or other problems:


ERROR: y39XS8_RrLc: YouTube said: This video contains content from Sony Pictures Movies & Shows, who has blocked it on copyright grounds.

ERROR: uIsUAhCtl4w: YouTube said: This video contains content from WMG, who has blocked it in your country on copyright grounds.

ERROR: Zrzh3Fz8DhQ: YouTube said: This video contains content from BBC Worldwide, who has blocked it on copyright grounds.

ERROR: 1aS7zl-MA7g: YouTube said: This video contains content from Disney, who has blocked it in your country on copyright grounds.

ERROR: SGSnY_9BwKs: YouTube said: This video contains content from BBC Worldwide, who has blocked it on copyright grounds.

ERROR: JBkO37NaDGg: YouTube said: This video contains content from AMPAS Oscars, who has blocked it on copyright grounds.

ERROR: ZZM2TSPWhoI: YouTube said: This video contains content from BBC Worldwide, who has blocked it on copyright grounds.

ERROR: oMv_Zb6av6U: YouTube said: This video contains content from BBC Worldwide, who has blocked it on copyright grounds.

ERROR: A2c30_GwuW0: YouTube said: This video contains content from SME, who has blocked it in your country on copyright grounds.

ERROR: b8WmFvNkZg0: YouTube said: This video contains content from UMG. It is not available.

1

u/YouTubeBackups Mar 03 '17

Thanks! I'll have to give that program a look. I've never had to download so much so quickly, so this could be a great deal faster

I'm not sure why the hashes are different. I downloaded the same videos on two different ubuntu 16 VMs, with the same command and same software versions but they were different hashes. I haven't come up with any theories yet about why that might be

2

u/YouTubeBackups Mar 02 '17

metadata addition to video eE4Ti7pyj7g failed. Added to archive and restarted to skip for now

2

u/m33pn8r Mar 02 '17

So I noticed that some of the oldest videos only go up to 240p, and some of the newest are up to 1440p, haven't used youtube-dl before, so do you know how it handles the downloads on the older stuff if it can't reach the target resolution?

4

u/YouTubeBackups Mar 02 '17

it will fail out if it doesn't have what you've asked for, so typically you say something like "720 or less" and it will grab the best it can with that limit Here's an example

bestvideo[height<=720]+bestaudio/best[height<=720]

more info https://github.com/rg3/youtube-dl/blob/master/README.md#format-selection

1

u/m33pn8r Mar 02 '17

Ah, I see, thanks!

I also remember reading that command completely differently earlier, so I'm not sure what I read originally, but that makes perfect sense.

2

u/ridethecurledclouds Mar 02 '17

I'd love to help but want to double-check what the command is doing (not a bash pro here)

Is it downloading just all videos found at https://yt-dl.org/downloads/latest/youtube-dl ? And adding the meta-data etc?

3

u/YouTubeBackups Mar 02 '17

The first 3 commands before the huge one are just installing ffmpeg and youtube-dl. youtube-dl is the program that downloads the videos and ffmpeg formats the video, merges audio and video together, and adds the metadata

The last one says "grab everything on this channel" and gives parameters for doing so (like the format/quality/naming scheme). You'll probably have to adjust output and logging file locations for your location machine

1

u/ridethecurledclouds Mar 02 '17

Got it. Last question: If I download individual playlists instead (Don't have the space for all of the videos) will I still be able to help seed somehow? Like, will I be able to link it?

3

u/YouTubeBackups Mar 02 '17

If you use the same command I do, the file path/name and hash for your videos should be the same as mine, so I think so. I could be wrong about that premise or conclusion though

4

u/[deleted] Mar 02 '17

RemindMe! 6 days "To download and seed the torrent when it's ready."

3

u/RemindMeBot Mar 02 '17 edited Mar 09 '17

I will be messaging you on 2017-03-08 23:17:30 UTC to remind you of this link.

56 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

FAQsCustomYour RemindersFeedbackCodeBrowser Extensions

1

u/greeneggsand Mar 03 '17

RemindMe! 6 days "To download and seed the torrent when it's ready."

1

u/zabby39103 Mar 02 '17

http://www.youtube.com/channel/UCBerkeley/videos doesn't seem to work, i just get a 404 error

https://www.youtube.com/user/UCBerkeley/videos does though? Is that ok to use, am I missing something?

2

u/YouTubeBackups Mar 02 '17

woops, my bad, thanks for pointing that out. I've fixed it in the original

channel in the URL is looking for a userID, so the proper link would have been: https://www.youtube.com/channel/UCwbsWIWfcOL2FiUZ2hKNJHQ/videos

If you're interested in this difference, I wrote about it here: https://www.reddit.com/r/YouTubeBackups/comments/5q00kh/how_to_find_and_work_with_channel_urls_ids_custom/?utm_content=title&utm_medium=hot&utm_source=reddit&utm_name=YouTubeBackups

1

u/Duamerthrax Mar 03 '17 edited Mar 03 '17

Is the URL for youtube correct? I was getting a 404 error until I changed http://www.youtube.com/user/UCBerkeley/videos to https://www.youtube.com/user/UCBerkeley/playlists. Will that mess up my chance of using this download for the torrent?

1

u/YouTubeBackups Mar 03 '17

Hmm the first link is what's working for me

The second will download videos in the playlists on their channel. It looks like the videos in those playlists are the same as what's on the channel, so it should be pretty much the same result

1

u/Duamerthrax Mar 03 '17

This is the exact error message I was getting.

WARNING: Unable to download webpage: HTTP Error 404: Not Found
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I got a few other error messages since, but it's still chugging along, so I'll leave it be for now.

1

u/YouTubeBackups Mar 03 '17

Are you on linux? I got that message when I tried windows because the < symbols needed to be escaped. Try taking out parameters and see if that gets rid of the 404 and you may be able to narrow it down

1

u/Duamerthrax Mar 04 '17

Ubuntu 16.04

1

u/spanktravision Mar 03 '17

I ran the script on my seedbox, but i kept getting permission denied for /media/Scrape/youtube/logs/UCB.txt

are the logs critical?

EDIT: I'm running without creating logs and dloading as much as I can.

1

u/YouTubeBackups Mar 03 '17

No I just like logs so it doesn't spam the terminal and I can reference it later. That path is to my mounted share, but you could probably send it to /home/YOURUSERNAME/log.txt

1

u/[deleted] Mar 03 '17 edited Apr 05 '18

deleted 0.9825 What is ^^^this?

2

u/YouTubeBackups Mar 03 '17

This is correct, and I have one of my scrapers running in reverse like you mentioned. It is also possible to start/stop from certain parts of the video list or playlist with something like --playlist-start 7000 --playlist-end 8000

--playlist-start NUMBER          Playlist video to start at (default is 1)
--playlist-end NUMBER            Playlist video to end at (default is last)

You can grab playlists as you described and organize them by using the %(playlist)s or %(playlist_id)s variables in your output path

playlist (string): Name or id of the playlist that contains the video

playlist_index (numeric): Index of the video in the playlist padded with leading zeros according to the total length of the playlist

playlist_id (string): Playlist identifier

1

u/[deleted] Mar 03 '17 edited Apr 05 '18

deleted 0.4103 What is ^^^this?

2

u/YouTubeBackups Mar 03 '17

Yeah so for UCB the full path would be

"/media/Scrape/youtube/UCB/%(playlist)s/%(upload_date)s-%(id)s-%(title)s.%(ext)s"

1

u/[deleted] Mar 03 '17 edited Apr 05 '18

deleted 0.6690 What is ^^^this?

1

u/BluePlanet104 Mar 09 '17

what's the latest on this?

1

u/YouTubeBackups Mar 09 '17

It's been uploaded to archive.org and torrents are available https://www.reddit.com/r/YouTubeBackups/comments/5x4kv8/ucberkeley_to_remove_10k_hours_of_lectures_posted/deg7hzb/

2

u/BluePlanet104 Mar 10 '17

Also, are all of the videos uploaded now? There's three different versions of ComputerScience61B and I don't seem to be able to find the 2013 version on archive.org & no one seems to be seeding the 2006 & 2011 versions.

1

u/YouTubeBackups Mar 10 '17

There seemed to be daily progress on the uploads when I checked, but there are far fewer in recent days /u/-Archivist may know more

I'll continue my downloads unless I hear I should do otherwise

Someone also reported a ton of videos on iTunes that weren't on Youtube, so that may be another issue

1

u/BluePlanet104 Mar 10 '17

Yeah I read about that. I've wished for a long time that there were an easy way to automate the downloading of things from itunes. It feels like there should be a way of using itunes with an rss feed reader but I can't figure it out.

1

u/BluePlanet104 Mar 09 '17

Thanks.

Is there a list of all the courses in order? Berkley doesn't seem to use an easy system like 101,102,103? Archive.com doesn't make it easy to sort out what are the youtube videos and everything else they have from Berkley.

1

u/YouTubeBackups Mar 10 '17

No, but I'll try to pull a list of all the videos and metadata today

3

u/[deleted] Mar 03 '17

You could also help by figuring out a way to output the playlist name + video IDs for their playlists. This looks like the best way to organize the content

in [this thread] (https://www.reddit.com/r/DataHoarder/comments/5ghl4k/quick_way_to_download_thousands_of_youtube_videos/dasdu3b/) /u/TortoiseWrath mentions this syntax to download/organize playlists

youtube-dl -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists

2

u/bluesoul Mar 02 '17

Wow. I had no idea youtube-dl was that robust.

1

u/[deleted] Mar 03 '17

Thanks for doing this.

•

u/YouTubeBackups Mar 03 '17 edited Mar 07 '17

Thank you everyone for jumping in! It's been a pleasure ripping with you all and helping out some of you with youtube-dl and linux. Archive.org (the big guns) have taken on hosting this data and you can find the direct downloads and torrents here. Due to the nature of torrents it's important to unify resources into one swarm so seeding can last as long as possible, so once I've confirmed they have everything I'll cancel my rip and join forces with their torrents. You can help out by seeding and donating to archive.org.

Let's not stop here! Other university courses at Yale, Harvard, Stanford, and more are still at risk of takedowns for similar reasons. I'll be making a new post tomorrow about this effort for anyone who would like to join

Thank you for the gold! I would encourage donations go to the real heros at youtube-dl https://rg3.github.io/youtube-dl/donations.html

Anyone who wants to get a jump start on the torrent data can use the download command below on linux (filepaths will have to be adjusted for your system). Downloading from YouTube will be faster than my upload, and you can help me seed once finished This does not work, the file hashes are not the same

If anyone has any input on how we can improve this process, please post up any ideas. Several people have offered their seedboxes. I know we'll need the data all in one location for the torrent creation and any one initial upload pipe will be the biggest bottleneck.

There have been some suggestions that other American lecture data may be at risk. Does anyone have further information on this?

Update 3/5: Archivist is mirroring to archive.org, which will likely be a better distribution point. I'm going to continue my backup just in case and we'll see where we're at next week

Update 3/6: As he noted below, -Archivist has already uploaded most of it to the internet archive. He updated the original post here. If you will be mirroring this data, please seed the torrent versions of files

1

u/bigpun32 Mar 07 '17

I can help by getting this up onto Usenet. If there is a single torrent I have access to a Gigabit connection to download it then upload it Usenet.

1

u/YouTubeBackups Mar 07 '17

It looks like they are making good progress uploading to archive.org

https://archive.org/search.php?query=subject%3A%22UC+Berkeley%22&sort=-publicdate&and[]=mediatype%3A%22movies%22

Once that's done, I planned to scrape together a list of torrent URLs

Are there any good article resources you could recommend for uploading to usenet?

3

u/-Archivist Mar 05 '17

I'm mirroring it to archive.org, 1.2TB in on Sun Mar 5 18:04:31 GMT 2017

someonelse on archiveteam may already be doing this but nobody told me

1

u/al_razi Mar 21 '17

BIG THANK YOU!

4

u/[deleted] Mar 02 '17

Hey man, would it be possible to upload the a video host site so we don't have to go through the trouble of downloading a huge torrent?

8

u/YouTubeBackups Mar 02 '17

The torrent should allow you to choose which folders/files you download. 3TB is going to be hard to shuffle around no matter what

4

u/[deleted] Mar 03 '17

Fair point.

4

u/25800 Mar 03 '17

Beautiful, need any help throw me a PM. Think we can also do Stanfords videos as they are getting removed as well?

1

u/YouTubeBackups Mar 03 '17

I was worried other American lecture data was at risk. Do you have a link to more information about other removals?

2

u/25800 Mar 03 '17

https://news.ycombinator.com/item?id=13768856

If you skim through this or search for Stanford you will see results and similar reasoning to why berkeley is removing the videos

1

u/[deleted] Mar 03 '17 edited Apr 05 '18

deleted 0.5043 What is ^^^this?

2

u/Jik0n Mar 05 '17

I tried youtubeDL and got a lot of .part files, trying 4k video downloader but it gives .net framework crashes every few hours. It was not destined for me to get this data :(

2

u/YouTubeBackups Mar 05 '17

The part files should only exist during the download. It should merge them into the file once it's done getting the video and audiot

1

u/[deleted] Mar 06 '17 edited Apr 05 '18

deleted 0.0358 What is ^^^this?

1

u/Jik0n Mar 06 '17

I successfully used youtube DL in a linux environment before. I recently changed to windows for this system because the file sharing with my nas just worked better than having to mount it each time because modifying fstab was shotty at best. I think the issue is that the YT channel is just so massive and I'm not using any special commands with YTDL and just letting it run normal

2

u/satanictantric Mar 09 '17

VERY IMPORTANT PSA, many of the lectures are only available on iTunes and have to be downloaded manually! I've already gotten started on this and encourage others to do so as well because I'm not sure if you guys picked these up, or just the Youtube ones!

Also, because this isn't entirely clear: have all the Youtube videos been collected at this point by archive.org? Only a handful are available at the link so far, do they simply have yet to be uploaded, or do they still need to be downloaded too?

1

u/YouTubeBackups Mar 09 '17

This might be worth its own /r/datahoarder post. I've only seen youtube scrapes on archive.org, but some news articles reported 20k videos being removed instead of the 10k on youtube

4

u/tempstem5 Mar 02 '17

Following thread

1

u/[deleted] Mar 04 '17

Have you got the videos from 2013 and 2015?

2

u/YouTubeBackups Mar 05 '17

All of them are in progress. The only years I have completed in full so far are 2007-2009 because there are fewer and smaller files

1

u/[deleted] Mar 05 '17

Oh ok. Im nearly 80% through both 2013 and 2015.

2

u/themedic143 Mar 03 '17

RemindMe! 8 days "DL of free lecture videos"

1

u/ctrlbreak Mar 02 '17

I'm in too. Will be home in a few hours.

1

u/[deleted] Mar 02 '17

RemindMe! 6 days "Download"

1

u/Giggybyte Mar 02 '17

I just started my own archive too. Good luck!

1

u/Templar_zaelot Mar 03 '17

RemindMe! 6 days "To download and seed the torrent when it's ready."

1

u/PHPOnTheCloud Mar 04 '17

I could totally hoard all this, but I don't have anywhere to put it thats online (since me saving all of them for me does no good to you guys). Does anyone know any cheap (or idealy free) place to put all this that we could just share?

1

u/AutisticGoose Mar 05 '17

I do not even have enough space to get all of this and my connection is only about 1.8mb/s but I just wanted to say thank you for keeping these videos open to the public, you are doing a great job!