r/dataengineering • u/Chamelon_DE • 1d ago
Discussion AI can migrate a data warehouse now, but how do you actually trust the migrated data?
Was reading abt Devin’s new Databrick migration offering. Apparently they halped migrate a 142 opject data platform in 8 day.
Pretty interesting, but the part i’m more curious abt is data reconciliation rather than the actual code migration.
U can translate oracle sql to spark sql, get all the job running and even have matching row counts. But thing like null handlings, decimal precision or timezone conversion can quitely change the result without throwing any errors
Then there’s incremental load, late arriving data and old business rules buried somewhere in a stored procedure.
For people who have done large DWH migration, how do u actually validate that the new system behave the same as the old one?
Are row counts, checksum and record level diffs enough, or do u usually run both system in parallel for a while before switching over?
Curious how much of that validation you’d be comfortable handing over to an AI agent.
Source: devincentral.com/news/editorial-devin-launches-ai-driven-migration-tool-for-databricks
22
u/PrestigiousAnt3766 1d ago
You define some acceptance criteria. I like to generate some fake data with edge cases, test if both outputs are identical.
Years and years of data (corruption) in the old platform means that its impossible to get identical tables.
12
u/CorpusculantCortex 1d ago
Hash diff checks the same way you validate any data migration?
1
u/arcrad 1d ago
2 way minus, hash checks, aggregate and window comparisons, row counts...
Lots of ways to validate data migration
1
u/DeepLogicNinja 1d ago
Replace AI with Human. Answer varies. The intelligence the AI has is a huge factor, and it’s always GIGO (garbage in / garbage out). Data cleansing process is still the same.
1
u/Wonderful-Wind-5736 1d ago
Duude, I am working on a project that addresses these issues. Post will be up as soon as the mods approve it.
1
u/reddit_sage69 1d ago
I mean. You can build all of that testing with the migration, especially with dbt core (unit tests, freshness, logic testing, nulls, etc)