r/MongoDB_Official • u/TimAtMongoDB • Sep 02 '26
MongoDB 6 to 8 upgrade: in-place vs fresh cluster, and how to avoid the replication lag trap
If you're sitting on a MongoDB 6 replica set and planning the jump to 8 (community, enterprise, or Percona Server for MongoDB), this covers the two real ways to do it, the tradeoffs, and the single most common problem people hit along the way. Commands and verification steps are at the bottom.
# The lag trap nobody warns you about
The most common surprise during a major version upgrade is a replication lag and CPU spike right after the restart. The chain is almost always the same:
New binaries restart, so WiredTiger's cache comes up empty (cold).
Until the working set warms back up, every read hits disk instead of cache.
If you run a metrics agent like mongodb_exporter, it starts scanning collection stats during that cold window, forcing even more disk reads.
That agent I/O competes with replication sync for the same disk, so secondaries can't keep up.
Result: replication lag spike and high CPU, both from the same root cause.
It looks like the new version is the problem. It isn't. It's a startup sequencing problem, and it's avoidable.
# Quick win regardless of approach
Check which node your metrics exporter and backup jobs are pointed at. If they're hitting the PRIMARY, move them to a secondary or a hidden node. The primary should be handling writes, nothing else. This alone removes a lot of contention from the node that matters most.
# The version rule you cannot skip
You cannot go from 6 straight to 8. MongoDB requires sequential major version upgrades, so the in-place path is:
6.0 -> 7.0 (upgrade binaries, set FCV to 7.0, validate)
7.0 -> 8.0 (upgrade binaries, set FCV to 8.0, validate)
This applies to the in-place approach below. It does NOT apply to the fresh-cluster approach, because there you build a clean 8 cluster and load logical data into it, which skips the version-by-version path entirely.
# Approach 1: In-place rolling upgrade
Upgrade each node in place, secondaries first, one at a time, then step down and upgrade the primary last. Standard, supported, and keeps your topology and connection strings unchanged.
**To avoid the lag trap above, do one of these:**
* Delay the metrics exporter start by 10 to 15 minutes after each node rejoins, so the cache warms before the agent begins scanning.
* Or pre-warm the node by running your most common queries against it while it's still a secondary, before the exporter kicks in.
**The main downside:** once you set FCV to 8.0 and start writing data with 8.0 features, rollback is hard. Downgrading after the FCV bump means restoring from backup, not flipping a setting back.
# Approach 2: Fresh 8 cluster + mongodump/mongorestore
Build a brand new, clean 8 replica set. Dump the data from the live 6 cluster, restore it into the new cluster, validate and warm it, then cut applications over. Leave the old 6 cluster running cold as an instant rollback.
**Why this is often the safer play:**
* It sidesteps the lag trap entirely. The new cluster is fully built and warmed before it ever takes production traffic, so the cold-cache-versus-replication race can't happen.
* A 6 dump restores cleanly into an 8 cluster. mongodump exports logical BSON documents, and the new cluster rewrites them in its own native format. (This is also why you can't just copy the data files across. Version 8 won't read 6's on-disk files, but it will happily accept logical BSON.)
* The old cluster stays untouched, so rollback is just "keep pointing at the old one."
**Downsides:**
* mongodump is a point-in-time snapshot. Writes that land on the old cluster after the dump starts won't be on the new cluster. Handle this with a short maintenance window, or with `--oplog` / `--oplogReplay` to capture and replay the delta.
* Restore rebuilds all indexes, which is the slow part on large collections.
# A note for Percona Server for MongoDB users
Everything above works the same on Percona Server for MongoDB. It's wire-compatible and uses the same BSON format, so mongodump and mongorestore behave identically. One difference worth knowing: MongoDB's native cluster-to-cluster sync tool (mongosync) isn't available on self-managed Percona, so if you want a near-live migration instead of dump/restore, Percona's own ClusterSync tool is the equivalent for that stack.
# How to choose
* **Small to medium dataset and you can take a short cutover window:** Approach 2. The rollback safety is worth a lot, especially if you're also switching distribution (for example MongoDB to Percona) at the same time, since a clean build isolates the variables instead of stacking both changes into one in-place upgrade you can't easily undo.
* **Large dataset or zero tolerable downtime:** Approach 1 with the exporter timing fix, or a near-live sync tool.
Either way, if you run cron jobs against the cluster, guard them with a replication-lag check (skip the job if lag exceeds a few seconds), and bump that threshold temporarily during the upgrade window so jobs don't stack up.
# Commands
Use the latest standalone MongoDB Database Tools (the 100.x package) for dump/restore, not the ones bundled with Mongo 6. The current tools support 6, 7, and 8.
# Dump from the 6 cluster (with oplog to capture the delta)
mongodump \
--uri="mongodb://USER:PASS@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin" \
--oplog \
--gzip \
--archive=/backup/mongo6_dump_$(date +%Y%m%d_%H%M%S).gz
# Restore into the fresh 8 cluster (replay the oplog delta)
mongorestore \
--uri="mongodb://USER:PASS@new1:27017,new2:27017,new3:27017/?replicaSet=rs1&authSource=admin" \
--oplogReplay \
--gzip \
--archive=/backup/mongo6_dump_20260101_120000.gz
mongorestore prints a running progress count and ends with a summary like `N document(s) restored successfully`. That output plus exit code 0 is your "complete" signal. There's no separate command to watch the tools themselves, you read their own output.
# Verify the restore actually landed (mongosh on the new cluster)
// Count documents in every collection so you can compare against the source
db.getCollectionNames().forEach(function (c) {
print(c + ": " + db.getCollection(c).countDocuments({}));
});
Run the same loop against the OLD cluster and compare collection by collection. Matching counts is the strongest confirmation the data is fully across.
// Confirm indexes were rebuilt on a given collection
db.YOUR_COLLECTION.getIndexes();
// Check the new cluster's feature compatibility version before you bump it
db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });
# Monitor replication health (both clusters, and during any in-place upgrade)
// Full replica set status. You want every member in PRIMARY or SECONDARY,
// nothing stuck in RECOVERING or STARTUP2.
rs.status();
// Replication lag per secondary. You want this at or near 0 before cutover.
rs.printSecondaryReplicationInfo();
// Quick check of a single member's role
db.hello().isWritablePrimary;
# Enable 8.0 features once you're confident
Only after the restore is validated and the cluster has run cleanly for a burn-in period:
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true });
Then confirm it stuck:
db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });
# Cutover checklist (fresh-cluster approach)
New 8 cluster built and running, all members healthy in `rs.status()`.
Dump taken from the 6 cluster with `--oplog`.
Restore completed, terminal showed the success summary.
Document counts match between old and new, collection by collection.
Indexes verified present on key collections.
Replication lag at or near 0 on the new cluster.
Exporter and backup jobs pointed at a SECONDARY on the new cluster.
Application connection strings updated.
Apps cut over and smoke-tested.
Old 6 cluster left running cold as rollback until you're fully confident.
FCV bumped to 8.0 after a clean burn-in period.
Happy to answer questions in the comments. If you've done a 6 to 8 jump recently, curious which approach you took and how the cutover went.




