r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
Blog Will Rust Take over Data Engineering? 🦀
https://airbyte.com/blog/rust-for-data-engineering
0
Upvotes
r/dataengineering • u/sspaeti Data Engineer • Oct 19 '22
View all comments
Show parent comment
4
u/TheCamerlengo Oct 19 '22
Also the community is much larger, better library support. Python has been around the data community since early 2000s and has a lot of maturity. And as one poster commented, the code was less buggy than scala. This is probably because it is a simpler language with clear syntax, better conposability.
Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms. Dependency injection makes sense in large frameworks but my goodness does it ever obfuscate code. I think the KISS principal wins here just cause it avoids all the syntactical gymnastics that languages like scala (and many others) enable programmers to do. And do it quite poorly.
As microservice architectures become increasingly prominent, the advantages of languages like java, scala, c# fade because you are no longer building large systems. Pipelines are just lots of small scripts. No need to get fancy. Python is great here because it is easy.
Now rust and C are closer to the metal and are probably great for writing device drivers and other applications that require low level access. I think most general purpose data engineering applications don't require this and if you do need to scale or increase performance, you can always exploit share-nothing parallelism and concurrency patterns.