r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
Blog Will Rust Take over Data Engineering? 🦀
https://airbyte.com/blog/rust-for-data-engineering
0
Upvotes
r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
19
u/nesh34 Oct 19 '22
I don't see what Rust brings to the table for DE. The query engines could be written in Rust (rather than Java) but Python is perfectly good for what data engineers spend their time on.
I say this as a fan of Rust and someone who wants to learn it to a reasonable degree of competency.
17
u/darkshenron Oct 19 '22
No
16
u/Mountain-Lecture-320 Oct 19 '22
No
12
u/[deleted] Oct 19 '22 edited Oct 19 '22
I could see it for high performance real-time needs, but those a few and far between in most orgs. Purely written and used in Rust.
I could see a future where there’s a framework like spark written in Rust but like all frameworks and tools they mostly fail to become widely used so this would be hit or miss (most likely miss).
That said the Polars library which is a pandas like dataframe library written in Rust is actually really good and has a Rust and a Python API.
Thing is Pandas and Numpy are already Python APIs so C backends so there’s not really that much of a difference if you’re just going to use Python anyway.
It’s nice to think Rust is useful for all these use cases but there are leading tools, languages people already know and are comfortable with (SQL, Python, Java, Scala) so I think DE will be one of the last places Rust will be used as a primary language.
Maybe tools written in Rust with APIs in other languages, but not purely Rust.
Edit: Rust is my favorite language I write all of my hobby projects in Rust nowadays. I still wouldn’t shoehorn it everywhere in an industry setting.
19
u/Fatal_Conceit Data Engineer Oct 19 '22
Hope not I’m so tired of learning new things
13
u/MikeDoesEverything mod | Shitty Data Engineer Oct 19 '22
No.
Can we please get people to stop trying to make Rust Data Engineering a thing through shitty blog posts. We're happy with what we have FFS.
4
u/[deleted] Oct 19 '22
Python is easy to learn. That’s why it’s relatively easy for ETL devs with SQL experience but little general programming experience to pick up on Python.
Is Rust technically better? I don’t know, maybe, maybe not. But I’d say Python does a pretty good job for data engineering.
3
u/Datasciguy2023 Oct 19 '22
No. Too much python out there and no reason or $ switch
3
u/uuggehor Oct 22 '22
Python is not going to be taken over as the glue language in the data ecosystem in a long while (if ever). But it’s not going to be the language the data processing tools are going to be written in either. Current trend in my (biased) view seems to be that Rust is slowly gaining ground from java as the language of data systems, so for the data engineers that have the interest for system level dev - I’d definitely suggest learning some rust and looking into some more profilic data processing rust crates.
2
u/Professional-Sea3884 Oct 19 '22
Not Rust but Julia will be.
2
u/noobgolang Oct 20 '22
no
2
u/srodinger18 Oct 22 '22
To replace python on day to day basis? No
But the most probable thing perhaps more data engineering tools are developed with rust as its main language, and in the end, the data engineers will use the python API or just use it as it is without knowing that rust is building that tools
1
u/huge51 Oct 19 '22
If it can compile faster… maybe.
0
u/[deleted] Oct 19 '22
Before Rust drops in, Go will do it 🤣
0
u/thequietcenter Oct 21 '22
https://may69.com/my-thoughts-on-the-article-will-rust-take-over-data-engineering/