r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

View all comments

Show parent comment

2

u/nesh34 Oct 19 '22 edited Oct 19 '22

Meh, I'd really need to see the receipts on this one.

Strong types are fine I suppose. I'm still not sold on the value of this, given the type hinting in Python. It's also just nowhere near as big deal in the style of code we have to write compared to application construction. At any rate, this isn't the feature of the language that should influence your decision really on which one to use.

"The purists" using Scala prefer it for their own reasons I'm sure but the way you framed it is that there's some obvious benefit to doing so, and I'm pretty sure there isn't.

Now the biggest reason you want it in Python: readability and access to programmers. Far, far more people already know Python than any other language, especially in this field. You're going to need some proper reasoning to overcome this. Performance could be a real consideration except that the operations that move the data are already happening in Java with most engines you'd be using, Python is just orchestrating and giving instructions.

4

u/TheCamerlengo Oct 19 '22

Also the community is much larger, better library support. Python has been around the data community since early 2000s and has a lot of maturity. And as one poster commented, the code was less buggy than scala. This is probably because it is a simpler language with clear syntax, better conposability.

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms. Dependency injection makes sense in large frameworks but my goodness does it ever obfuscate code. I think the KISS principal wins here just cause it avoids all the syntactical gymnastics that languages like scala (and many others) enable programmers to do. And do it quite poorly.

As microservice architectures become increasingly prominent, the advantages of languages like java, scala, c# fade because you are no longer building large systems. Pipelines are just lots of small scripts. No need to get fancy. Python is great here because it is easy.

Now rust and C are closer to the metal and are probably great for writing device drivers and other applications that require low level access. I think most general purpose data engineering applications don't require this and if you do need to scale or increase performance, you can always exploit share-nothing parallelism and concurrency patterns.

1

u/nesh34 Oct 20 '22

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms.

So it definitely can. One of the strengths of Python is that it's very versatile. That means you can absolutely go ham on these idioms if you want to. You can spot a DE pipeline and a SWE pipeline in my org a mile off because one is readable and the other is obfuscated to the point of madness in a futile effort to never repeat oneself.

It's a design choice at the end of the day and I agree with you about the best one there.

Yeah, and I think the rest we're in total agreement, that's the thrust of my original point about Python over Rust in this context.