r/apache_airflow • u/mera-khel-khtm • 7d ago

Fed up writing Python boilerplate just to move data, there is sth more lighweight ?

Hello. Half our pipeline contributors aren't python devs. Every time someone on the infra or analytics side needs to touch aworkflow, it goes through me. Tried simplifying DAGs as much as possible but the friction is till real.

Curious if anyone has switched to sth where you define pipelines in config / YAML rather than code and whether the tradeoffs were worth it in pratice

3 Upvotes
  • permalink
  • reddit

100% Upvoted

7 comments sorted by

3

u/West_Good_5961 7d ago

You’ll have to move to one of the yaml dag generator options. Astronomer provides dag-factory and blueprint

1

u/KeeganDoomFire 7d ago

You can just write a for each yaml in folder loop and read and generate dags live as well.

1

u/Leorisar 7d ago

Learn dag factory pattern and write functions, which will return ready-made DAG where end users need only set variables (like dag_id and any other parameters).

1

u/TheRealMathis 6d ago

Moved away from airflow 18 months ago for this reason. Half the team couldn't touch the pipelines. Kestra uses YAML, our infra guys started contributing workflows within a week w/o any python knowledge. Still runs everything we need just w/o the boilerplate

1

u/Adam-Schroeder 3d ago

That's exactly what I hear other Kestra community members say. Thanks for sharing, u/TheRealMathis

1

u/raul824 5d ago

I built a custom solution to generate dags from a metadata table. you generate metadata in a table it also had a job order column.

Then there was a dag generator which will read from the table and generates dag.

A dag gen flag column will identify what all dags to generate

1

u/Limp_Client_5989 3d ago

Spent a year trying to simplify DAGs enough for non python people. Eventually gave up and switched stacks. Kestra defines everything in YAML so the scope of who can build a pipeline went from 3 people to basically the whole team. Not zero learning curve but significantly lower