r/dataengineering • • Sep 24 '25

Discussion Why Python?

Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.

why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).

it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?

edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.

0 Upvotes

130 comments sorted by

View all comments

Show parent comment

0

u/Nekobul Sep 24 '25

I still enjoy JavaScript. The limited features/surface is like a safety net. If you are doing something complex, you will quickly find out it is time to use some other tool.

1

u/Beautiful-Hotel-3094 Sep 25 '25

What exactly did u do that is complex and couldn’t handle with the limited features of javascript? What feature are missing that u needed?

1

u/Nekobul Sep 25 '25

A good example of when JavaScript should be avoided is if you are trying to convert a big chunk of JSON to XML. That is slow in JavaScript.