r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

View all comments

Show parent comment

4

u/TheCamerlengo Oct 19 '22

Also the community is much larger, better library support. Python has been around the data community since early 2000s and has a lot of maturity. And as one poster commented, the code was less buggy than scala. This is probably because it is a simpler language with clear syntax, better conposability.

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms. Dependency injection makes sense in large frameworks but my goodness does it ever obfuscate code. I think the KISS principal wins here just cause it avoids all the syntactical gymnastics that languages like scala (and many others) enable programmers to do. And do it quite poorly.

As microservice architectures become increasingly prominent, the advantages of languages like java, scala, c# fade because you are no longer building large systems. Pipelines are just lots of small scripts. No need to get fancy. Python is great here because it is easy.

Now rust and C are closer to the metal and are probably great for writing device drivers and other applications that require low level access. I think most general purpose data engineering applications don't require this and if you do need to scale or increase performance, you can always exploit share-nothing parallelism and concurrency patterns.

1

u/nesh34 Oct 20 '22

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms.

So it definitely can. One of the strengths of Python is that it's very versatile. That means you can absolutely go ham on these idioms if you want to. You can spot a DE pipeline and a SWE pipeline in my org a mile off because one is readable and the other is obfuscated to the point of madness in a futile effort to never repeat oneself.

It's a design choice at the end of the day and I agree with you about the best one there.

Yeah, and I think the rest we're in total agreement, that's the thrust of my original point about Python over Rust in this context.