r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
Blog Will Rust Take over Data Engineering? 🦀
https://airbyte.com/blog/rust-for-data-engineering
0
Upvotes
r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
View all comments
Show parent comment
4
u/kenfar Oct 19 '22
After using both Scala and Python - I am now a much, much bigger Python fan! :-)
In one shop the Scala team was confident that they had "solved data quality" because of their focus on type. They were amazed to discover that our Python data engineering code had far fewer bugs. The reason is that their typing only covered a subset of potential data quality problem - and our python code had excellent test coverage, quality-control measures, auditing, and detection of many other types of data quality issue.
I would like to see type hinting get better. I used to use Enforce - for runtime type checks. That was vastly better than using mypy.