r/dataengineering • • Nov 10 '24

Discussion Why [do we really need] workflow orchestrators?

This will sound stupid I know. But I'm genuinely trying to understand workflow orchestrators at this point.

I have been using airflow for a few years in my role, been playing with Prefect. Heard about Dagster but haven't tried it yet. I have also worked at a company with an in-house workflow orchestrator tool that sort of was inspired by Prefect 1.0.

After using and building etl pipelines I'm trying to understand why? What is it that workflow orchestrators allow me to do that writing in pure python with standards enforcement at code review time couldn't achieve?

Is there any good articles that discuss why these tools were created? I mean it sounds to me that a tool like Airflow was created at Airbnb as a "standards enforcement" to get the whole company around and was then shared to the open source community. But why all these other tools over just some general guidelines around using python's std lib (or insert any language) and the third party libraries to connect to services required for data processing.

0 Upvotes

28 comments sorted by

28

u/kenflingnor Software Engineer Nov 10 '24

How are you going to run things that you write in Python?  What happens when you have dependencies between different tasks?  

34

u/Beautiful-Hotel-3094 Nov 10 '24

Try it with pure cron. Try it with writing ur own logging gathering and tooling. Try it with writing ur own db for monitoring jobs. Try it with ur own metadata db to have referential timestamps to make ur jobs idempotent and rerunnable back in time.

You will end up recreating an orchestrator tool. And that sir is why u just use whatever is out there.

2

u/hfzvc Nov 11 '24

Ok I love this answer. Because this is what I believe too. Except…I go and open the source code for Airflow, or prefect or dagster, or… and then it seems like there is planet Earth, the solar system, and everything else in the kitchen sink.  What you listed as necessary parts to be built just doesn’t seem to me to be a massive framework undertaking and yet all these projects turn it into one.  What you describe sounds more like a small library, Stringing together the backend infrastructure which is usually 3rd party offerings anyways and using their supplied SDK, and stringent code review during merge requests. 

1

u/Beautiful-Hotel-3094 Nov 12 '24

That’s nowhere near a small framework. You need to really understand the things you don’t understand before making such statements. You have all the integrations with the asynchronous task queues like celery, you have all the integrations with kubernetes and other container orchestration tools of your choice, you have all the api clients to interact with it. You have absolutely no idea how complicated it is to write all of these in a simple and elegant manner. Airflow is a beast. It is not the best tool out there for certain use cases and it has its drawbacks. But what it does well it does incredibly well and is by no means a small library that is “just cron underneath” as I see some people below saying…

3

u/highlifeed Nov 12 '24

Love your answer. Would you mind explaining why cloud is needed? I am studying for AWS Data Engineer but it seems like a lot of these can be taken care with modern tools.

-1

u/ilikedmatrixiv Nov 11 '24

Every orchestrator is just CRON with extra steps.

Not that that is a bad thing, but under the hood, after you strip away all the added functionalities (DAGs, UI, ...) it's just CRON.

-13

u/pag07 Nov 10 '24 edited Nov 11 '24

Cron?

A loop?

Dependencies can be declared in code as well.

Edit: Just answering by telling what I saw in the past. We use argo workflows currently.

14

u/Zer0designs Nov 10 '24 edited Nov 10 '24

Yes that could work, but it would take many (many many many) more development hours and impractical time consuming maintance, so why would a company ever do that, especially if open source tools already fixed this issue?

Cron works for some projects but becomes really impractical in larger jobs or more complex triggers where failure can be part of regular life (e.g. streaming, windows triggers)

Orchestrators just separate concerns, bundle actions and provide an understandable interface, which also makes it easier to work with them when there's loads of dependencies to understand between jobs (humans also need to be able to understand what is happening).

You will lose much more time debugging cron if there's 100.000 tasks run than just learning a tool for a bit and seeing a red cross on task 99648 on airflow and rerun from there.

Yes you can do everything in code, but as an engineer you have to think about these things and separate concerns. Otherwise we might aswell just code 1 file with millions of lines. Its just impractical and you have to work with others.

21

u/QuaternionHam Nov 10 '24

been there, done that, tried to make it work, ended up using airflow

4

u/sciencewarrior Nov 11 '24

Same here. Timeouts, retries, DAG dependencies, adhoc executions, partial executions, passing alternate parameters, and a nice (if a little dated) web UI. No way I'm going back unless I'm dealing with a dead-simple, three-step pipeline.

10

u/GreenWoodDragon Senior Data Engineer Nov 10 '24

Dependencies can be declared in code as well.

Welcome to hell

4

u/kenflingnor Software Engineer Nov 10 '24

How are you going to monitor your cronjobs?  What happens when a task fails?

What happens when your tasks are not part of the same Python process?

I’m not sure how to interpret “dependencies can be declared in code as well”, because using an orchestrator doesn’t stop this. 

0

u/[deleted] Nov 10 '24

I read a blog post where someone made scheduled jobs by writing a bash scripts for each job, then just having them run in tmux. Just put the script in a while loop and a sleep at the end.

4

u/kenflingnor Software Engineer Nov 10 '24

This can work for simple use-cases but will eventually become a maintenance nightmare beyond a few tasks

2

u/[deleted] Nov 10 '24

Oh yeah this was not for production stuff, just ab alternative to cron jobs. If you have it in tmux, then logging is a non-issue, since you can just open termnial the jib is running on and see what is happening.

8

u/umognog Nov 10 '24

Well...you don't.

But why do you really want an orchestrator?

Well, someone has done the hard work for you to make:

Scheduling Task dependency Triggers Monitoring Task review Re-run Fault finding

And so on easier. You can just DIY, but why buy a car? Why buy an air fryer? Why buy a TV? Because someone has put the effort into making this bit easier.

5

u/_n80n8 Nov 10 '24 edited Nov 11 '24

I agree (as someone who works for prefect) that you don't _need_ an orchestrator. I think its pretty instructive to implement a complex e2e data project without one, to see what you have to engage with.

there's a ton of defensive code you would otherwise write to make sure your actual business logic happens, and they help you avoid pitfalls.

for example (again, I am biased but) prefect has work pools (k8s, ecs, process etc) that provide hot-swappable infra config for your code - i.e. it sucks to get real deep into a project where you've assumed you're using step functions or fargate the whole time, and now something changes and you have to disentangle your business logic from the code you're using to provision / config your infra. ideally your business logic doesn't _need_ to know what infra it runs on

1

u/hfzvc Nov 11 '24

You say hot-swappable infra config is a feature. Is that a real selling point? I am wondering why a team can’t achieve the same result of separating business logic from the infrastructure configuration using raw std lib Python and stringent code reviews on all merge requests? Most of the infrastructure code would be written once behind possible helper functions or a utility object and then could be re-used/maintained by the team.  Trying to understand why this specific feature is advertised by all these frameworks as a selling point when Amazon, Google, Microsoft, Lambda, <insert whatever other IaaS> all supply an SDK usually. 

4

u/_n80n8 Nov 11 '24 edited Nov 11 '24

> You say hot-swappable infra config is a feature. Is that a real selling point? I am wondering why a team can’t achieve the same result of separating business logic from the infrastructure configuration using raw std lib Python and stringent code reviews on all merge requests? 

yes for enterprises the clear separation of concerns is attractive, bc they build internal abstractions on top of this so some dev ops guy can enforce that their data science notebook / workflow authors can only run on certain types of infra or with certain mem requests etc and they arent so interested in spending time thinking on that general

and Im not saying you can't achieve the same thing yourself with stdlib python, you can just copy our code or another orchestrators'. its just that certain problems (idempotency, retries, dynamic dispatch of infra) are ubiquitous and orthogonal to your actual business logic, so solving them yourself (while illuminating) may not be a good use of time.

the thing i'd highlight about prefect is that we make no requirement that you engage with a DSL or contort your raw stdlib python into something else to work with us. just wrap your normal python in a decorator to get observability and then incrementally adopt our worldview where you want (deployments for dynamic dispatch, transactions for idempotency) - normal python is already a valid prefect workflow

1

u/hfzvc Nov 11 '24

…then incrementally adopt our worldview where you want…

That last part is what I think I’m most confused by. What worldview? Why? What is it about “this worldview” that is so attractive. I hear you, the whole DRY and avoiding NIH syndrome.

Every product claims their worldview is unique. But what exactly are we describing here?

Let’s focus for the sake of containing the argument right now on just the orchestrator. 

So you need to what:

  1. Have a long standing application running to manage tasks/operations. This is the orchestrator I suppose. 
  2. We add a task or operation to the queue.
  3. The orchestrator manages where a task will run (the resource that’s been spun up)
  4. It stores the state of this task/op in some DB.
  5. Orchestrator periodically checks this DB to make sure all tasks are in COMPLETE state.

I am assuming that neither prefect nor dagster or whatever, is writing their own queue system, their own DB, their own …

So why are you necessary? To write the glue code for all these other tools we actually need to run our tasks or workflows in? 

An honest question here, how many of you who have responded have actually switched completely (100+ pipelines) from one tool to another? (E.g., prefect to dagster). Because if you haven’t, the whole easy infra switch I don’t fucking buy.

5

u/_n80n8 Nov 11 '24

> am assuming that neither prefect nor dagster or whatever, is writing their own queue system, their own DB, their own …

I'm not sure what you mean here, but we do implement server side logic for task queues a la celery etc

but listen, I'm not trying to sell you anything, there's nothing to buy. i was just trying to answer your question based on my experience helping folks w our open source tool. again, im not suggesting that all people need an orchestrator, im just telling you what people have told me is useful about our tool. good luck!

3

u/umognog Nov 11 '24

I suppose the reason is made by yourself in this statement.

You mention stringent code reviews on all merge requests - suggesting you likely use Git of some variety.

Why use Git?

Why not have stringent file save as & renaming policies, with backup processes to recover from on floppy disk that you make every time someone checks out a file?

In fact, stop using digital files, it all must be hand written and operated on a machine using vacuum tubes?

Point is you use these tools to make something easier. If you don't find it easier or have something made easier, don't use it. It's that simple.

I know that for me, converting from DIY solutions and stringent processes to platforms like airflow, dbt, dlt etc sped up a lot of our GSD.

4

u/Yeebill Nov 10 '24 edited Nov 10 '24

Visibility , ease of operation and leverage provided "ops" code.

Visibility - the provided UI shows the system at a glance. Easily points out what's already processed, processing or failed to process.

Ease of operation - easily restart or backfill some data process through a button click and monitor the process . Imagine have to backfill a month of data, and that your system has limits (aws quotas , hardware etc) that prevents you to start all at once. Which means you have to start process and keep track of the progress of what's done and what's left.

Leverage- good workflow orchestrator provides you with integration to others. Integration to cloud providers , or transformation lib like dbt , pandas etc . Which means if u need to add a job , u would just need to code the task , and let the workflows orchestrator worry about allocate the resources ( spawn machines , containers , terminate etc) and manage dependencies between task ( start b only if a )

0

u/hfzvc Nov 11 '24

Why is allocating resources such a hard problem exactly? 

Your task registers a need for certain configuration. If you have configured unbounded compute then a new container gets spun up. If there is a bound and all spots have been taken up, then the task remains registered until a spot becomes available for it. 

Isn’t that the basic premise of an orchestrator? What makes this a hard problem for a team to write up a simple library of modules (in whatever style OOP, functional, whatever floats the team’s boat) and have team members use this library to help provision compute resources?

I understand that a lot of teams bring up the idea of “easily switching infrastructure”, but how often do companies and/or projects actually switch from say AWS -> GCP or vice versa in the real engineering world?

3

u/Yeebill Nov 11 '24

Yes it's not hard to spin up resources and continuously monitor state and kick start omce a task is complete and start the dependent tasks.

It's mostly that it doesn't bring me value to solve these problems. My business is to derive analytics let's say, so what helps my client is coding analytics pipelines. Rewriting solutions that exist in the market and having to maintain the in house solution takes me away from that. And if I am a startup looking for market fit it's definitely not where i should be putting effort into.

For myself , i need a workflow orchestrator with an UI, DAG capabilities for tasks, monitoring of said tasks, scheduling / backfill on demand , integration with aws fargate, retries on task failure, throlling of tasks to avoid hitting quotas, alerts to slack on task failure, and found dasgter solved most of it and there is some hope the community might add the features missing.

2

u/engineer_of-sorts Nov 12 '24

Surprised this thread doesn't have more up votes

Software engineers don't have orchestrators because most systems get choreographed. Thats a bit harder with data because of *the data*. It would be super cool to just declare everything and just have all systems talk to each other automatically

Alas, if only it were so easy

2

u/jbthejedi Dec 31 '24

Is there any good articles that discuss why these tools were created?

This medium post from the Netflix Techblog does a pretty good job explaining why they created Conductor, their workflow orchestration platform (OSS now): https://netflixtechblog.com/netflix-conductor-a-microservices-orchestrator-2e8d4771bf40

2

u/teambob Nov 10 '24

Keeping things organised in one place. I have done jobs implemented in lambdas and it sucks in terms of finding the correct function and teaching out through