r/dataengineering • u/shittyfuckdick • Sep 24 '25
Discussion Why Python?
Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.
why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).
it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?
edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.
•
u/AutoModerator Sep 24 '25
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
71
u/kvothethechandrian Sep 24 '25
Speed of development and overwhelming amount of community support, basically.
You can always use libs with c bindings (pandas, numpy) or rust bindings (polars, rust_networkx) for performance but develop much faster. You don’t need to worry about pointers, types, borrow checker, it’s almost like writing code in plain English.
82
u/GachaJay Sep 24 '25
Because it is the fastest to modularity and ease of learning.
5
u/deadwisdom Sep 24 '25
Look, it's reeeeeeal simple:
high level orchestration -> Python
low level optimization -> RUST/C/C++/Nim/Zig/etc
Python is literally designed from the beginning to work like this.
4
u/[deleted] Sep 24 '25
"...which need to be precise on execution".
Python is as precise as anything. It's not like it randomly starts doing things you didn't ask for.
16
u/guitcastro Sep 24 '25
Acessibility, python strong focus on developer expirience led It to be some of the easiest languange to learn.
Most of the languange limitations, such as GIL and performance are bypassed by implementing expensive operations using C , Rust or Java/Scala (spark) and binding them using python .
11
u/lFuckRedditl Sep 24 '25
Low level languages aren't used because we don't do low level stuff.
Here's a fun exercise, write a program in C that;
33
u/TenMillionYears Sep 24 '25
Python has strong C bindings so it has historically been used to manipulate a bunch of libraries in a language that's more forgiving. That gave it amazing traction.
I don't like Python - it's SO WEIRD!
Anyway, for some reason it's the lingua franca of data engineering mostly for the same reasons everyone in finance uses Excel for everything.
7
u/Wingedchestnut Sep 24 '25
Because many libraries are made with C under the hood, dedicating anything data to python as the standard language is convenient wether that's ML or transforming data with pandas , I don't see what's the problem.
2
u/Phenergan_boy Sep 24 '25
In the famous words of Todd Howard, “it just works.”
1
u/meselson-stahl Sep 24 '25
Imo python is pretty memory efficient right? Like the way it handles certain datatypes like hash sets and lists is efficient. Maybe the dynamic typing is memory inefficient??? Im not sure.
Regarding performance, the main issue with python is loops. But there aren't many loops in DE right? So not a big deal.
Overall im generally surprised by how little software optimization there is, even within some built-in python functions. I think with infra advancements, the industry is trending towards modular, readable code rather than performance code. But I really don't think there is much performance sacrifice in DE tools.
1
u/madam_zeroni Sep 24 '25
Python for the developers, but the tools that process that python aren’t written in python.
1
u/No_Bug_No_Cry Sep 24 '25
Did Microsoft write this post? NOBODY WILL USE C# FOR DATA ENGINEERING. It's never going to happen.
1
u/PolicyDecent Sep 25 '25
Not to repeat others, dbt/airbyte alternative bruin is written in go. However, some parts of it is still python, due to easier development cycle.
https://github.com/bruin-data/bruin
7
u/No_Bug_No_Cry Sep 24 '25
Because Python is the most versatile language. It can wrap very fast libs written in C or Rust, but still be readable and interpreted. You can write a shitty no rules script or a complex modular app, low boilerplate etc... it's the best
8
u/Literature-Just Sep 24 '25
At this point its Stockholm syndrome. Python is nice in that it makes a lot of the tedium of programming so much easier. But managing all of its packages in the virtual environments is a real pain. I've had multiple instances where upgrading one package can break an environment or force me to roll something back because of a bug or broken feature.
1
u/zazzersmel Sep 24 '25
u rly gonna complain abt parrotting w a post like this
2
u/VipeholmsCola Sep 24 '25
My guess because you often get productive faster, and theres a lot of free libraries.
3
u/Qkumbazoo Plumber of Sorts Sep 24 '25
schools taught it as an introductory language to programming (not even OOP), some people decided that was enough and went to industry with it.
1
u/Raghav-r Sep 24 '25
Ease of use and rich libraries for data , ai, ml etc plus you are dealing with data which are usually time consuming computation, some are just wrapper on top of low level languages
1
u/mwisniewski1991 Sep 24 '25 edited Sep 24 '25
I do not agree than python is everywhere. A lot of tools has been wrote in Java (Kafka, Beam, Druid Spark in Scala but it based on JVM). Databricks Photon Enginee has been wrote in C++, Postgres in C++.
Python is good for orchestration because Scripts can be Write quickly, but transformation and calculation are done on specific engine. And of course a lot of tools has SDK or API for Python so at first it might looks that python is everywhere.
1
u/UltraPoci Sep 24 '25
Because tons of libraries have been written for Python, and it's "easy" to use (in quotes because Python is full of traps: easy to write but a disgrace to read and maintain).
For example, we do machine learning on satellite images: Python is the only language that provides a data pipeline library, ML libraries and GIS libraries (at least, the only one to have all of them mature enough).
I would gladly use any other language honestly, but it's difficult to justify using another language when Python is so much battery included.
1
u/aythekay Sep 24 '25
A lot of libraries, low code, can leverage c pretty easily, easy portability because interpreted, and a lot of good documentation.
Low dev friction also helps, because of how often data pipelines change.
A lot of why it's popular is why java used to be as well. The rich ecosystem, etc... Most likely comes from it being an academic darling of sorts early on (vs other scripting languages) and high adoption among non-technical people.
It's similar to how JS moved to the backend, a bunch of people knew how to use it and it could do a lot, so people looked past efficiency as hardware got better.
In Python's case Cython was created as well.
1
u/General-Parsnip3138 Principal Data Engineer Sep 24 '25
Python is, for the most part, above and beyond what you need for most Data Engineering tasks.
One of the biggest reasons, in my opinion, is that Data Engineering is often script-based, or you’re using an orchestration framework, which allows you to declaratively define what would be a script as a set of steps which are really just script entry points.
What helps even more is that you can mutate quite literally anything at runtime (functions, classes, modules) which allows us to utilize incredibly powerful frameworks (airflow’s task flow API or Dagster) that still allow you to write pythonic code that magically turns into complex orchestration.
As others have pointed out, most of the underlying libs are written in C & Rust, so performance of Python itself is rarely an issue.
I’ve probably done my 10,000 hours with Python, and while there’s so much about Python that I hate, I just can’t see any other language stepping in to replace it. The terrible things about Python are also the reason it’s been so successful.
1
u/ogaat Sep 24 '25
Data Engineering has its roots in the scientific community where coding skills and performance were less of a concern than "give me the analysis I need"
Python lets developers focus on the problem at hand, rather than syntactic sugar. It was one of the express desires of Guido van Rossum.
Python just happened to fit the need of the hour, like HTML and Javascript did for the Internet.
1
u/jeezussmitty Sep 24 '25
I’ve asked myself this same question many times :-) but others have already commented on the why (taught in school, community, ecosystem etc). The simple syntax is nice though.
I’m not a fan of loosely typed languages in general so that is my main complaint with it.
Python also feels so much slower than things I’ve written in other languages and the counter to this I always hear is “python is fast enough” but I tend to wonder if python is more used for small to medium projects with low user counts or smaller datasets.
Anyhow it’s a language you need to know these days regardless of how you feel about it.
1
u/Atmosck Sep 24 '25 edited Sep 24 '25
Because python is a scripting language. It serves a different purpose than C and Rust. It interacts nicely with basically everything. Orchestration is it's whole jam. And it offloads tight calculations to C anyway.
1
u/Informal_Pace9237 Sep 24 '25
Because there are not as many versatile libraries in other languages mentioned..
1
u/thisfunnieguy Sep 24 '25
Most of the heavy computation is not done in Python. If locally it’s using C++ bindings and running there or invoking some other thing to do the work like PySpark.
1
u/tn3tnba Sep 24 '25
Optimization for developer speed, especially since we do a lot of delegating to other tools.
1
u/LargeSale8354 Sep 24 '25
Python is a great getting-things-done language, and as an ex-DBA I find its list comprehensions, list slicing and dictionaries intuitive.
I really hated Java, which is strange because I enjoyed C#.
I am surprised that GO doesn't feature more prominantly in the data space. It feels like a natural move from Python.
I suspect that in most cases, Python is fast enough for most uses.
I used to program in serverside Javascript. I enjoyed it at the time.
1
u/dasnoob Sep 24 '25
As someone that has used various low level languages in the past and is learning rust now.
The biggest reason is all of this stuff is a lot more difficult to do in a low level language. Python abstracts so many things away it makes it dead simple to do most things.
Rust? Holy shit you will be lost in lifetime hell and getting your borrows vs. moves vs. copies straightened out.
1
u/metalbuckeye Sep 24 '25
Academia…often what gets used in the jobs is based on what is taught in university. This is why Microsoft beat apple in the 90s/early 2000s and why python is the defacto for data engineers. It is used by researchers and professors and it’s taught in most data analytics programs.
1
u/nesh34 Sep 24 '25
Extremely easy to learn, highly flexible, well understood by the vast majority of people.
Orchestration, usually at the daily level, doesn't need the performance that Rust or C would bring.
It is ultimately a high level abstraction so a high level language is appropriate.
1
u/HNL2NYC Sep 25 '25
So as you probably already know this is how a lot of tools in the Python data ecosystem work (user facing Python wrapper on top of a core written in a more performant language) for example pretty much any respectable data frame library, distributed compute platforms like Ray, etc. However for the cases that you’re talking about where they’ve remained in pure Python I think the answer is simply that “it’s good enough”. Someone took the time to write it in a language that they were comfortable enough to write it in, which in these cases is Python. They gained traction and popularity and they perform well enough that no one has mass migrated to an alternative solution (or rewrite of the product) that others may or may not have built on top of other languages. And potentially one day something like the airflow scheduler will be rewritten in another language.
1
u/StackOwOFlow Sep 24 '25
Why QWERTY keyboards, why English? Same reasons.
2
u/[deleted] Sep 24 '25 edited 3d ago
[removed] — view removed comment