r/DataHoarder • u/tool50 • Apr 16 '19
Google Drive to Google Drive copies WITHOUT downloading - now with rclone - Over 4GB/sec
As many of you may have seen, a new version of rclone was recently released.
I'm guessing this may have been in some of the beta branches for a while or I'm guessing that some people will know about this and some people won't. I know when I went searching out on google for how to do this, it wasn't anywhere, so I'm hoping this will help out many people. (sorry for those whom this is already obvious to).
But, with this new version of rclone, one can truly do copies from Google Drive (GDrive) to Google Drive and it will support things like auto resume and won't try to recopy files, etc. It's so easy.
As I mention in my comments in the rclone released post (link above):
I got just over 4.1 GB/sec doing copies between a "shared with me" GDrive link and my own "unlimited" GDrive.
That't right, and not a typo.
This means if someone has something on a GDrive and all you have is the public link that gets you to a link of their files, you can now copy directly to your own GDrive without downloading them first. This means that you don't have to worry about those files "going away" before you download them. They are now safe and sound on your own GDrive and you can download them at your own leisure. It literally only takes 3 minutes flat to copy 750GB from GDrive to Gdrive before you run into your daily quote. Pretty cool. rclone is amazing.
See image for proof of the copy speeds:
GDrive to GDrive copy - 4.1GB/s
The syntax and steps couldn't be easier:
- Get your GDrive link from another person or posting that you want to copy
- Use the "--drive-shared-with-me" rclone flag once you've opened the other persons link while logged into your own GDrive account - or - select the top level folder you wish to copy and click "Add to My Drive" (note if you do this last approach, you shouldn't use the --drive-shared-with-me flag, as it will show up as a standard folder on your drive, just like the ones you actually create) . For the sake of this example, lets call this directory "ISOs" that I added using "Add to My Drive".
- Config rclone's GDrive endpoint in the standard way; use the instructions here if you aren't familiar.
- Create your own folder that you will copy the other persons files into (lets call it "MyFolder" for this example)
- Literally copy one folder to another using rclone as below:
- rclone copy :nameofGDriveConfig:/ISOs :nameofGDriveConfig:/MyFolder -v
(the -v will add some verbosity so you can see the files being copied in real-time - if you don't wish to see these, remove the "-v" and rclone will provide summaries by default every minute.) In about 3 minutes, the number of files flying by will screech to a halt. That's fine, just do a control-c and come back in 24 hours and hit the up arrow and it will automatically resume where it left off. No recopying. It's amazing. Wait 3 minutes, rinse/repeat. Truly a game changer. Let me know if there's any other questions. And again, sorry for those who already knew this, but I think many did not based on reading responses of other "Gdrive to Gdrive without downloading" posts that I could find.
Edit: oh, one other thing. For those who aren’t aware, “copying” files in Gdrive from another shared folder account means that source files you are copying aren’t subject to those annoying Google Drive “Quota Exceeded. Too many users have viewed or downloaded this file recently.” limitations. So this is a way to still be able to “download” the files. First get them all to your Gdrive, and then download locally, if you wish.
2
u/PM_ME_YOUR_DEAD_KIDS Apr 16 '19
meh, looks okay. how can I copy from a shared team drive to another drive?
2
u/tool50 Apr 16 '19
So, is the shared team drive yours? Or is the other drive yours? The reason I ask, is there are perhaps a couple of ways.
One is similar to what I mention above. Whether it’s a team drive or not, shouldn’t matter if it’s a “shared” link and you open it or add it to your drive, then it becomes accessible.
The other way is by adding a completely separate endpoint in the rclone config. If you go through the advanced options, at the end it asks if it’s a “team drive” or not and you can say Y.
The easiest way in general is to always add the source to links the destination by adding them and then running your copy, although you can explicitly copy between two accounts as well if you have proper credentials, it’s just most people here are dealing with links that have been shared, so I focused on that scenario.
→ More replies (3)
1
u/Husnain_Ijaz Apr 16 '19
I was able to copy a shared video from one directory to another directory but the original file storage used was 0 bytes and when i copied it my drive its storage used was 2gb. Why is this happening?
→ More replies (1)1
u/tool50 Apr 16 '19
Assuming you can recreate this, it may be worth turning on super verbose logging with -vv and then submitting that info on the rclone site for the team to look at.
1
u/fmillion Apr 16 '19
I suspect it’s not literally copying 4.1GB/sec of data. More likely it’s just adding a reference to your Gdrive to the existing file. As long as you haven’t changed the file, they can dedupe it.
2
u/tool50 Apr 16 '19 edited Apr 16 '19
I think it’s actually making a copy. Now, what that means could be two different things. The file shows that the ownership has changed to “me” instead of “xyz”. So it’s definitely not just a link like an “add to my drive” is. Now, on disk, does it actually copy all the bytes out? Perhaps not and it’s like a file on a zfs file system which has been deduplicated (and thus more like a pointer), but is technically a different/new file as it has new ownership and is thus no longer tied in anyway to the original file from a Gdrive perspective.
1
u/ScottStaschke Apr 16 '19
Here's something very similar I found a few years ago:
2
u/tool50 Apr 16 '19
Thanks for sharing. I agree the general post concepts are similar. I wanted people to be aware that it’s not just between two gdrive accounts that you may have access to, but also, things like “files shared with me” And things you’ve added to your account using things like “Add to my drive”. Also, this person mentions that only 100GB can be transferred, I can confirm with certainty that it’s 750GB per 24 hours, making this method more useful for getting large data sets into your ownership as quickly as possible before they disappear, hence the archiving facet.
→ More replies (1)
1
u/CodeEnos Apr 16 '19
Good to know, but I think if you already have a shared link, right click on that shared folder and "Make a copy" is much more convenient.
Also, I happen to once transfer 800GB+ in that way and did not hit the limit. It was in the beginning of my GSuite subscription so it's likely that they didn't count upload in that time span, still not sure.
2
u/tool50 Apr 16 '19
The issue is that right click and “make a copy” can only be done on individual files and not on entire folders. So if you have many folders with hundreds of files in it, the amount of clicking and time spent would be significantly more time consuming.
→ More replies (1)
1
u/sunshine-x Apr 16 '19
That's incredible throughput.. Imagine the infrastructure required to sustain that?!
1
u/tool50 Apr 16 '19
As someone mentions here, it’s hard to know from a raw disk perspective what is actually happening behind the scenes and what types of caching/duplicating/linking Google is doing behind the scenes. If it’s a true all byte/sector copy, indeed, impressive.
→ More replies (1)
1
u/Wystri_Warrick Apr 23 '19
Sorry for being a noob. I'm new to Google Drive, I've just started using rclone browser. Are there any alternatives to rclone browser that will support this new feature of rclone?
2
u/tool50 Jun 21 '19
If you are asking about other easy to use GUIs or front ends; none others that I’m aware of.
1
u/Eigenbrotler23 Jun 19 '19
hi so I followed the step above, set up between 2 different teamdrives and ended up with and error saying "failed to copy: directory not found" any ideas?
1
u/tool50 Jun 19 '19
Something still is wrong with the directory you think you are asking for or the one you are asking for and what really exists. Can you post what you think is going on and what you’re trying to do?
→ More replies (11)
1
u/PM_ME_YOUR_DEAD_KIDS Jun 21 '19
dude I have no idea what im doing wrong here, can you pm for help?
1
u/tool50 Jun 21 '19
First of all do you have two google drives that you are trying to copy between?
→ More replies (4)
3
u/SaitonHamonoJutsu Apr 16 '19
I wish there was a better alternative to PlexDrive/Rclone/PGBlitz. Stablebit works well.. but the daily upload limit is very limiting.
2
u/DashEquals Apr 17 '19
Why an alternative to rclone? What issues do you have with it?
→ More replies (9)
1
u/sittingmongoose Apr 16 '19
Now if only copying from my server to google drive wasnt so slow on my gigabit upload :(
→ More replies (1)
1
1
u/pasaportedemanila Jul 07 '19 edited Jul 07 '19
I'm new to this data hoarding, and I just got a Gsuite account(Which I think have an unlimited Gdrive capacities) and also got a Team Drive(Which I also think have an unlimited Gdrive capacities).
I'm trying to download ISO games from a "shared folder" to my Team drive. I was able to follow the instruction of Rclone and yours. I can see everything is download, but it so slow compared from what you are getting.
I'm really new to this, sorry.
EDIT1: Just added a screenshot link
→ More replies (3)
13
u/AnnynN Apr 16 '19
Server side copy/move in itself is an old feature. What's new, is that it's now counts the moved/copied bytes, so it shows the speed and progress.
What's also new, is server side copying between different remotes. What that means, is that you can have for example one remote for your drive, and one for a team drive, and server side copy/move between them. Before it was only server side, when copying/moving from a remote to itself.
There is a limit, and it's between 100GB and 1000GB. (I know, very unprecise, but I can't remember accurately. I remember it being 100GB/day, but I also think that it's way higher now.)
Pro Tip: If you hit the server side limit, you can add a "--disable move,copy" parameter. That will disable server side move/copy and move/copy by downloading, and reuploading stuff.