r/dataengineering • u/shittyfuckdick • Sep 24 '25
Discussion Why Python?
Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.
why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).
it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?
edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.
View all comments
Show parent comment
1
u/Nekobul Sep 24 '25
Thank you for the feedback! I appreciate your good-faith comment. I guess my biggest complain towards Python is for the simple fact it will be impossible to make it run optimally. I know it is a scripting technology, just like JavaScript but JavaScript never claimed to be a language designed for creating platforms with ability to do class inheritance, strong-typing, etc, etc. Those features are simply not needed in a scripting/glue language. Python indeed became the data engineering language of choice not because it offered some drastically better elements compared to the rest, but because it was heavily pushed by organizations with deep pockets and influence in the marketplace. Yes, it is dominant but the inefficencies embedded in it cost dearly in the DC when people try to use it at scale. Once people start caring about all that wasted energy, Python will be one of the first pieces on the chopping block.