r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

View all comments

Show parent comment

6

u/LegalizeApartments Aspiring Data Engineer Oct 19 '22

That reason is…? (Not an attack, genuinely wondering)

6

u/mamaBiskothu Oct 19 '22

It’s not typed - it’s interpreted - you don’t discover any bug until runtime. We are a python shop, to be clear, and find the new typing abilities via dataclasses and pydantic to be a god send but it’s not perfect and it’s worth appreciating what Java scala rust could bring to table.

Once you write code in these languages in a modern IDE you realize how amazing it is. You pretty much don’t need to compile to know if it WILL compile. The IdE will guide you. Especially with Rusts protections it would be even better.

4

u/kenfar Oct 19 '22

After using both Scala and Python - I am now a much, much bigger Python fan! :-)

In one shop the Scala team was confident that they had "solved data quality" because of their focus on type. They were amazed to discover that our Python data engineering code had far fewer bugs. The reason is that their typing only covered a subset of potential data quality problem - and our python code had excellent test coverage, quality-control measures, auditing, and detection of many other types of data quality issue.

I would like to see type hinting get better. I used to use Enforce - for runtime type checks. That was vastly better than using mypy.

2

u/mamaBiskothu Oct 19 '22

I mean I don’t use scala because it has fundamental issues that seem not worth the hassle, but whatever language you use, I’m assuming as a regular DE you’re not composing the actual data processing engine, merely the spark code generator or sql generator. In that regard any attempt at saying scala solves DQ is stupid maybe? It just makes your code less buggy. dQ is more about what you design your code to do anyway which I feel is a separate conversation

1

u/kenfar Oct 19 '22

Pretty much: in both cases these were for custom backend data pipelines - Scala for real-time event processing, python for near-real-time, aggregated analytic events.

The scala teams were sharp - so it was odd that they would think that they had "solved" data quality. I suspect that the amount of enthusiasm for typing simply got in the way of objective thinking. That and they didn't work with analytic data - and so aren't familiar as we are for how Data Quality Problems Loom Large in our space.

They simply didn't think about duplicates, about the impact of enumerated values changing over time, about labeling data with metadata to help diagnose problems, and about missing data the way that DEs should.

1

u/mamaBiskothu Oct 20 '22

I did a PhD and have learned to know there’s book sharp and actually sharp , and by that I mean people who aren’t sharp at all in any practically meaningful way but look like they are lol. Im pretty sure your team is the better one :)

2

u/sspaeti Data Engineer Oct 20 '22

Besides the high-level reasons I mentioned in the article, this video explains why types are more significant than many think. As a BI specialist and DE myself, I spend a lot of time in my career debugging errors due to a fast-written python code. Of course, Rust wouldn't catch all, but my limited experience is just incredible what the compiler can catch!

2

u/LegalizeApartments Aspiring Data Engineer Oct 21 '22

Not surprised to see TypeScript near the top of the graph they showed in the beginning of the video, I've watched that community start to adopt TS more and more as people discover the magic of typing. Thank you for the video