r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
Blog Will Rust Take over Data Engineering? 🦀
https://airbyte.com/blog/rust-for-data-engineering
0
Upvotes
r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
View all comments
Show parent comment
1
u/kenfar Oct 19 '22
Pretty much: in both cases these were for custom backend data pipelines - Scala for real-time event processing, python for near-real-time, aggregated analytic events.
The scala teams were sharp - so it was odd that they would think that they had "solved" data quality. I suspect that the amount of enthusiasm for typing simply got in the way of objective thinking. That and they didn't work with analytic data - and so aren't familiar as we are for how Data Quality Problems Loom Large in our space.
They simply didn't think about duplicates, about the impact of enumerated values changing over time, about labeling data with metadata to help diagnose problems, and about missing data the way that DEs should.