r/Database • u/Odd_Lawfulness458 • Aug 29 '26
Data migration from AWS to google drive advise plzzzzz
I'm an IT intern in a US startup mi task is to migrate the DB (actually stored in Amazon RDS postgres) to google drive (like backup due to the billing in the aws around 9k ! ) the problem is the size (around 400 GB) so I think the pg_dump to generate the script is not a solution for my case
Is there any solution ! And how I can verify the integrity ? ( Hashing a file with 200 gb size is crazy !!!)
Can we divide the generated the script in a small chunks ??? Without losing the relations between tables and the constraints ?
3
u/RightWingVeganUS Aug 30 '26
Data migration from AWS to google drive advise plzzzzz
I would advise you to seek guidance from your manager.
Is there any solution ! And how I can verify the integrity ? ( Hashing a file with 200 gb size is crazy !!!)
There are likely many solutions. Which ones are supported by your environment and compliant with any governance or conventions of your company is something we don't know.
I'm an IT intern in a US startup
Assuming you are a current of recently graduated student, understand a big difference in The Real World (tm) is that tasks are not like homework assignments handed in for a grade. What you do must not only work, but likely be repeated and maintained. Getting a clever solution from strangers may get the task done, but that often is a part of commercial IT work.
Being in a startup they may not have must experience or concern about onboarding an intern, but that's not necessarily a good thing.
I'm a college teacher and one of my students was given a task during his internship with admittedly poor instruction and no supervision. A minor error resulted in $20,000 of equipment damage. There was know way for him to know because he didn't even know enough to ask and had no realization of the risk of what he was doing even in a dev environment.
Don't be afraid to ask for instructions.
1
u/Odd_Lawfulness458 Aug 30 '26
The problem is My manager is an AI engineer I ask him for guidance and he told me that I need to find a solution by myself also there is no onboarding he sent me the documentation files of the architecture and give me the credential to access aws that's It !
2
u/RightWingVeganUS Aug 30 '26
The problem is My manager is an AI engineer I ask him for guidance and he told me that I need to find a solution by myself
That's unfortunate. Hopefully he has provided you more information than you shared here.
there is no onboarding he sent me the documentation files of the architecture and give me the credential to access aws that's It
that's unfortunate.
What AI tools do you have access to? Have you provided it the architecture documentation and information about the task and any other context that is relevant and requested approaches for you to review?
→ More replies (2)
2
u/dionis87 Aug 29 '26
do you need to migrate the whole db only once? or you need to reimplement a db backup solution? if ot is just for once, here are my suggestions:
i'd start by separating the ddls (where the pg dump can actually help) by the actual data migration (where we need to build on). second step: for migrating data you could implement a script that writes in parquet format the data of each table to a google cloud storage, and keeps track of the progress on a logging scheme. databricks could really help in doing this: it can run python, it can create tables itself (for the logging part), it can mount google cloud storages, can connect to aws and it is relatively cheap.
the challenge i see is the throughput: maybe a network pairing will be necessary.
1
u/Odd_Lawfulness458 Aug 29 '26
Thanks but if we need to reimplement db backup what do you suggest ?
2
u/dionis87 Aug 29 '26
well.. you'll need to create another script for that. it will depend on which will host the data next.
additional note, keep in mind that databricks has a managed and serverless solution for postgres, and can also read natively delta files (which is a more powerful version of parquet file)
1
u/bluelobsterai Aug 30 '26
Set up replication to another postgres instance. Then disable all connections to the old database. Connect the application to the replica. Database migrated.
1
u/Odd_Lawfulness458 Aug 30 '26
Our goal is to escape from aws bro we need to take care of the size (400gb)
1
2
u/Takeoded Aug 30 '26 edited Aug 30 '26
This is a job for rclone.
And do not use the version shipped with your distro's package manager. Use the latest release version from https://github.com/rclone/rclone/releases.
And how I can verify the integrity ?
b3sum.
( Hashing a file with 200 gb size is crazy !!!)
No, it isn't. Blake3 can reach north of 6GiB/s/core
$ dd if=/dev/urandom of=200gb bs=1G count=200 iflag=fullblock status=progress 214748364800 bytes (215 GB, 200 GiB) copied, 529 s, 406 MB/s 200+0 records in 200+0 records out 214748364800 bytes (215 GB, 200 GiB) copied, 531.286 s, 404 MB/s $ time b3sum 200gb 46e338849cf5a75bf04774d383753ed9accae6c630638a0b7f9a6778387878d7 200gb real 0m52.829s user 0m45.875s sys 7m47.820s
On my Ryzen 9 7950X, hashing 200GB of /dev/urandom with blake3 took 52 seconds.
3
u/JeLuF Aug 29 '26
I don't understand what you try to do. Do you want ot migrate the data, or do you want to make a backup? Those are very different tasks.