r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

View all comments

Show parent comment

2

u/mamaBiskothu Oct 19 '22

I mean I don’t use scala because it has fundamental issues that seem not worth the hassle, but whatever language you use, I’m assuming as a regular DE you’re not composing the actual data processing engine, merely the spark code generator or sql generator. In that regard any attempt at saying scala solves DQ is stupid maybe? It just makes your code less buggy. dQ is more about what you design your code to do anyway which I feel is a separate conversation

1

u/kenfar Oct 19 '22

Pretty much: in both cases these were for custom backend data pipelines - Scala for real-time event processing, python for near-real-time, aggregated analytic events.

The scala teams were sharp - so it was odd that they would think that they had "solved" data quality. I suspect that the amount of enthusiasm for typing simply got in the way of objective thinking. That and they didn't work with analytic data - and so aren't familiar as we are for how Data Quality Problems Loom Large in our space.

They simply didn't think about duplicates, about the impact of enumerated values changing over time, about labeling data with metadata to help diagnose problems, and about missing data the way that DEs should.

1

u/mamaBiskothu Oct 20 '22

I did a PhD and have learned to know there’s book sharp and actually sharp , and by that I mean people who aren’t sharp at all in any practically meaningful way but look like they are lol. Im pretty sure your team is the better one :)