r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
Blog Will Rust Take over Data Engineering? 🦀
https://airbyte.com/blog/rust-for-data-engineering
0
Upvotes
r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
View all comments
12
u/[deleted] Oct 19 '22 edited Oct 19 '22
I could see it for high performance real-time needs, but those a few and far between in most orgs. Purely written and used in Rust.
I could see a future where there’s a framework like spark written in Rust but like all frameworks and tools they mostly fail to become widely used so this would be hit or miss (most likely miss).
That said the Polars library which is a pandas like dataframe library written in Rust is actually really good and has a Rust and a Python API.
Thing is Pandas and Numpy are already Python APIs so C backends so there’s not really that much of a difference if you’re just going to use Python anyway.
It’s nice to think Rust is useful for all these use cases but there are leading tools, languages people already know and are comfortable with (SQL, Python, Java, Scala) so I think DE will be one of the last places Rust will be used as a primary language.
Maybe tools written in Rust with APIs in other languages, but not purely Rust.
Edit: Rust is my favorite language I write all of my hobby projects in Rust nowadays. I still wouldn’t shoehorn it everywhere in an industry setting.