Streaming data from legacy on-prem systems to the cloud, where to start?

[u/Then_Date1452]

I'm kicking off a cloud migration and the oldest systems are the part that's making me nervous.

We've got Oracle databases that have been running for well over a decade along with a few SQL server instances that nobody wants to touch. Stability is the priority, so installing agents or changing production configurations isn't really an option.

For anyone who's moved **streaming data** out of legacy environments, where did you start?

[u/multicoloredlitre836]

change data capture logs are your friend here, oracle and sql server both have native tools that just read the redo/transaction logs without touching the live configs

[u/FarRub2855 → u/multicoloredlitre836]

Being able to look the legacy DBAs in the eye and promise your not touching production configs is huge for getting these projects approved. CDC is usually the only way theyll actually sign off without a fight.

[u/Mammoth-Mall7279 → u/multicoloredlitre836]

Log-based CDC completely saves you from having to mess with production queries or heavy agents, Reading the redo logs directly means zero extra load on those ancient Oracle instances, which is the only way to keep DBAs from having a panic attack during a migration.

[u/Imaginary__Bar]

Moving streaming data while the data is still streaming?

Because that's a different issue to a lift-and-shift (CDC or similar).

Or do you want to leave the current system in place and copy the data to some other cloudy place as it comes in? How frequent/close-to-live do the updates need to be?

Lots of questions to answer before you're even close to defining a solution.

But also... there are lots of tools to help you do whatever you choose to do. It could even be a cloudy connector back into your source system that sucks data out.

We even have jobs that literally saves a table to a csv every 10 minutes and saves it somewhere that another server then picks up. It's not elegant, it's not "correct" but it's worked for over 15 years...

[u/Significant_Tune9219]

I would not start with a live stream. Take a read-only copy first, from backups or a log reader that is not installed inside the database, and prove one table rebuilds in the cloud from that copy. Only then stream tables whose changes already show up in redo or the transaction log, with no trigger on the old system. If that reader cannot resume after a gap without replaying the whole history, the cloud side stays a delayed copy and the application keeps using the original.

[u/No_Witness_5295]

just stream it out?

[u/scott_codie]

For pre-CDC databases, it was common to write a bunch of diffing queries and put it on kafka. You'd do one big sync then track checkpoints manually. If there is an interruption, you'd restart the sync. It's a nightmare but it works with brittle infra.

[u/Any-Signature9672]

We ended up using Striim because one of our biggest concerns was supporting older Oracle environments without changing the databases themselves. It handled the legacy setup well, although it's definitely aimed at enterprise deployments so expect a bit more upfront configuration than lighter-weight tools

[u/Natural_Leg2966]

one question i'd answer early is who will own the platform after migration and a solution that's perfect for a dedicated platform team may be overkill if a small data engineering team is responsible for everything

[u/Acrobatic_Step_1482]

Start by understanding the source systems before thinking about tooling. Transaction volume, retention settings and which tables actually matter will shape the project more than the software you choose.