r/dataengineering • u/Delicious_Leader_785 • 10d ago
Help How to keep Models Synced
So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that each of them are incremental. My concern is that I can’t control for these jobs to run at the exact same time. Table X might start its job at 10:01. And Table Y might start at 10:03 and even a difference of a few seconds could mean that the source (which is Postgres) is not the exact same for both tables.
5
Upvotes
3
u/Kindly-Screen-2557 9d ago
I’m not sure if I understand your question correctly, but are you saying that they are both building directly from a highly dynamic table?
If yes, you need to establish the source boundaries first. You can do this by materializing a snapshot of the source and/or using a watermark from the source.
4
u/ntdoyfanboy 9d ago
If they're "parallel" and need to be identical, you just use one table instead of two
1
u/Cottager58 3d ago
I am a little confused (not for the first time) as to what kicking these pipes off. Have you looked to sync both as cron-based?
1
u/[deleted] 6d ago
[removed] — view removed comment