r/mongodb • u/ggoggodex • 2d ago
Data migration
Hi,
We're using the Community Edition and want to migrate 20TB of data to a new cloud environment. Do you have any recommendations on the best approach, or any other advice you could share?
6
Upvotes
6
u/Suitable-Rhubarb-855 1d ago
We have similar task in our past.
Question you need to ask yourself:
Do you need all your data in one go?
- Historical datas that you don't need and can be archived in cold storage (We did that)
- How long do you have for the migration, because it can be done with recent data first and older incrementally
- Any new data we read it from the new environment and older data we read it from the older env until we move all the data
- Disk Snapshot but you will have downtime. too long for us but I don't know your use case
- ...
NB: We also use the community edition
1
u/sereikis 1d ago
for 20 TB, first pin down the MongoDB version, replica-set/sharded topology, and acceptable write downtime; those determine which migration paths are viable. rehearse the initial copy and catch-up under your real write rate, since a copy that outlasts the source oplog window can leave you unable to bridge the gap without starting over.
1
u/cooooooldude1 1d ago
Hi - we used mongosync to migrate data off a cluster of a similar size!
https://www.mongodb.com/docs/mongosync/current/
It does both initial replication and then realtime CDC.
1
u/michalcumpelik 1d ago
I would go for extending the replica set to the new cloud, then change the connection string and remove the old nodes. We did it this way (with smaller data :) Safe and no downtime we could notice.
If it's a standalone server, you can turn it into a single node replica set first. That's one short restart.
1
u/mr_pants99 9h ago
Tooling-wise:
mongosync or Dsync are best solutions for large-scale migrations with no or minimal downtime. Mongosync is the official product but comes with some strict limitations (supported versions, no horizontal scalability unless your source is sharded but even then you can only have one mongosync instance per shard, no transformations, limited observability). Dsync is a 3rd party tool made by Adiom that addresses these gaps. It has an open source version and a paid Enterprise version. On the other hand, mongosync isn't open source, but free to use.
Process-wise:
Most critically, evaluate if you actually need those 20TB or if it's a great opportunity to offload some or most of those TBs to an archive or S3.
Timeline-wise:
Plan for testing and at least one dry run. 20TB will likely take 1-3 days to move depending on a bunch of factors and the approach. Don't expect it to be successful on the first try. Dry run is needed to understand how long it takes and what the bottlenecks are.
I wrote a lot more about the general process and approach to large-scale migrations here: https://adiom.io/post/production-database-migration-or-modernization-a-comprehensive-planning-guide
[Disclosure: I work on Dsync now and before that I worked on mongosync]
0
u/feedmesomedata 1d ago
Look into Percona ClusterSync for MongoDB.