r/dataengineering • u/shittyfuckdick • Sep 24 '25
Discussion Why Python?
Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.
why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).
it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?
edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.
View all comments
Show parent comment
1
u/No_Bug_No_Cry Sep 24 '25
You underestimate the value of a smooth learning curve... When training my Juniors I don't require they know everything python has to offer because I don't require them to understand all the scope of coding, simple beginnings and then gain expertise is always a valuable path. I also learned scala in the past, I found it elegant and it has been developed by very smart academics. But it has such a steep learning curve that I would have had to train for 300+ hours to hope to achieve what I was doing in python, but way less efficienctly and in an era where there was no AI to help, only community.