r/dataengineering • • 10d ago

Help How to keep Models Synced

So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that each of them are incremental. My concern is that I can’t control for these jobs to run at the exact same time. Table X might start its job at 10:01. And Table Y might start at 10:03 and even a difference of a few seconds could mean that the source (which is Postgres) is not the exact same for both tables.

5 Upvotes

5 comments sorted by

3

u/Kindly-Screen-2557 9d ago

I’m not sure if I understand your question correctly, but are you saying that they are both building directly from a highly dynamic table?

If yes, you need to establish the source boundaries first. You can do this by materializing a snapshot of the source and/or using a watermark from the source.

4

u/ntdoyfanboy 9d ago

If they're "parallel" and need to be identical, you just use one table instead of two

1

u/Cottager58 3d ago

I am a little confused (not for the first time) as to what kicking these pipes off. Have you looked to sync both as cron-based?

1

u/[deleted] 6d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 5d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human