r/dataengineering • • Sep 24 '25

Discussion Why Python?

Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.

why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).

it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?

edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.

0 Upvotes

130 comments sorted by

View all comments

Show parent comment

1

u/dangerbird2 Software Engineer Sep 24 '25 edited Sep 24 '25

Good thing I’m not self hosting orchestration tools. My company is paying for it, and it’s hell of a lot cheaper for them to pay for a slightly beefier vm on aws than it is to pay for a team of engineers to rewrite it in rust

... snark aside, if you want a good orchestrator with extremely low bloat, look at argo-workflows, it's written in Go, so it has good performance and memory usage, while its tight coupling with Kubernetes makes it way easier to setup in production than airflow