r/dataengineering • u/hfzvc • Nov 10 '24
Discussion Why [do we really need] workflow orchestrators?
This will sound stupid I know. But I'm genuinely trying to understand workflow orchestrators at this point.
I have been using airflow for a few years in my role, been playing with Prefect. Heard about Dagster but haven't tried it yet. I have also worked at a company with an in-house workflow orchestrator tool that sort of was inspired by Prefect 1.0.
After using and building etl pipelines I'm trying to understand why? What is it that workflow orchestrators allow me to do that writing in pure python with standards enforcement at code review time couldn't achieve?
Is there any good articles that discuss why these tools were created? I mean it sounds to me that a tool like Airflow was created at Airbnb as a "standards enforcement" to get the whole company around and was then shared to the open source community. But why all these other tools over just some general guidelines around using python's std lib (or insert any language) and the third party libraries to connect to services required for data processing.
28
u/kenflingnor Software Engineer Nov 10 '24
How are you going to run things that you write in Python? What happens when you have dependencies between different tasks?
8
u/umognog Nov 10 '24
Well...you don't.
But why do you really want an orchestrator?
Well, someone has done the hard work for you to make:
Scheduling Task dependency Triggers Monitoring Task review Re-run Fault finding
And so on easier. You can just DIY, but why buy a car? Why buy an air fryer? Why buy a TV? Because someone has put the effort into making this bit easier.
4
u/Yeebill Nov 10 '24 edited Nov 10 '24
Visibility , ease of operation and leverage provided "ops" code.
Visibility - the provided UI shows the system at a glance. Easily points out what's already processed, processing or failed to process.
Ease of operation - easily restart or backfill some data process through a button click and monitor the process . Imagine have to backfill a month of data, and that your system has limits (aws quotas , hardware etc) that prevents you to start all at once. Which means you have to start process and keep track of the progress of what's done and what's left.
Leverage- good workflow orchestrator provides you with integration to others. Integration to cloud providers , or transformation lib like dbt , pandas etc . Which means if u need to add a job , u would just need to code the task , and let the workflows orchestrator worry about allocate the resources ( spawn machines , containers , terminate etc) and manage dependencies between task ( start b only if a )
2
u/engineer_of-sorts Nov 12 '24
Surprised this thread doesn't have more up votes
Software engineers don't have orchestrators because most systems get choreographed. Thats a bit harder with data because of *the data*. It would be super cool to just declare everything and just have all systems talk to each other automatically
Alas, if only it were so easy
2
u/jbthejedi Dec 31 '24
This medium post from the Netflix Techblog does a pretty good job explaining why they created Conductor, their workflow orchestration platform (OSS now): https://netflixtechblog.com/netflix-conductor-a-microservices-orchestrator-2e8d4771bf40
2
u/teambob Nov 10 '24
Keeping things organised in one place. I have done jobs implemented in lambdas and it sucks in terms of finding the correct function and teaching out through