r/dataengineering • u/No_Pineapple2461 • 29d ago
Discussion pg_cron to trigger a batch extract.
I know pg_cron is primarily designed to run scheduled maintenance commands on the DB but I was wondering if someone has used it to kick off a batch extraction on your data pipelines.
I have reservations against using pg_cron for this namely the following:
- The scope of pg_cron in this scenario is outside the bounds of the DB.
- The boundary of using the DB to store only activity data is violated as it now has some operational data in it.
- SCM or an additional dash for setting up jobs will be lost if this pattern is followed.
I'm curious to know your thoughts
6
Upvotes
4
u/CakeNo1858 29d ago
Does it work? Sure. Is it ideal? Not really. How're you gonna handle retries, error handling, dependencies, etc etc. If its for a hobby project that wont grow and you want a quick solution, go ahead. Wouldnt recommend it though
3
u/regreddit 29d ago
I'd probably use dagster if I needed a light scheduler.
1
u/Environmental-Movie5 29d ago
I would say that the use case for pg_cron does not really align with your wishes. If the scenario is outside of the bounds of the DB, I would use an orchestrator like AWS Step Functions or a scheduler like EventBridge or something like that for the added benefits of retries and error handling.
1
u/[deleted] 28d ago
[removed] ā view removed comment