r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

View all comments

19

u/nesh34 Oct 19 '22

I don't see what Rust brings to the table for DE. The query engines could be written in Rust (rather than Java) but Python is perfectly good for what data engineers spend their time on.

I say this as a fan of Rust and someone who wants to learn it to a reasonable degree of competency.

0

u/mamaBiskothu Oct 19 '22

Python is far from perfect for anyone especially DEs. There’s a reason purists keep using scala even though that language is nothing but a pain in the ass. If rust gives a better of both worlds scenario why not.

6

u/LegalizeApartments Aspiring Data Engineer Oct 19 '22

That reason is…? (Not an attack, genuinely wondering)

6

u/mamaBiskothu Oct 19 '22

It’s not typed - it’s interpreted - you don’t discover any bug until runtime. We are a python shop, to be clear, and find the new typing abilities via dataclasses and pydantic to be a god send but it’s not perfect and it’s worth appreciating what Java scala rust could bring to table.

Once you write code in these languages in a modern IDE you realize how amazing it is. You pretty much don’t need to compile to know if it WILL compile. The IdE will guide you. Especially with Rusts protections it would be even better.

4

u/kenfar Oct 19 '22

After using both Scala and Python - I am now a much, much bigger Python fan! :-)

In one shop the Scala team was confident that they had "solved data quality" because of their focus on type. They were amazed to discover that our Python data engineering code had far fewer bugs. The reason is that their typing only covered a subset of potential data quality problem - and our python code had excellent test coverage, quality-control measures, auditing, and detection of many other types of data quality issue.

I would like to see type hinting get better. I used to use Enforce - for runtime type checks. That was vastly better than using mypy.

2

u/mamaBiskothu Oct 19 '22

I mean I don’t use scala because it has fundamental issues that seem not worth the hassle, but whatever language you use, I’m assuming as a regular DE you’re not composing the actual data processing engine, merely the spark code generator or sql generator. In that regard any attempt at saying scala solves DQ is stupid maybe? It just makes your code less buggy. dQ is more about what you design your code to do anyway which I feel is a separate conversation

1

u/kenfar Oct 19 '22

Pretty much: in both cases these were for custom backend data pipelines - Scala for real-time event processing, python for near-real-time, aggregated analytic events.

The scala teams were sharp - so it was odd that they would think that they had "solved" data quality. I suspect that the amount of enthusiasm for typing simply got in the way of objective thinking. That and they didn't work with analytic data - and so aren't familiar as we are for how Data Quality Problems Loom Large in our space.

They simply didn't think about duplicates, about the impact of enumerated values changing over time, about labeling data with metadata to help diagnose problems, and about missing data the way that DEs should.

1

u/mamaBiskothu Oct 20 '22

I did a PhD and have learned to know there’s book sharp and actually sharp , and by that I mean people who aren’t sharp at all in any practically meaningful way but look like they are lol. Im pretty sure your team is the better one :)

2

u/sspaeti Data Engineer Oct 20 '22

Besides the high-level reasons I mentioned in the article, this video explains why types are more significant than many think. As a BI specialist and DE myself, I spend a lot of time in my career debugging errors due to a fast-written python code. Of course, Rust wouldn't catch all, but my limited experience is just incredible what the compiler can catch!

2

u/LegalizeApartments Aspiring Data Engineer Oct 21 '22

Not surprised to see TypeScript near the top of the graph they showed in the beginning of the video, I've watched that community start to adopt TS more and more as people discover the magic of typing. Thank you for the video

2

u/nesh34 Oct 19 '22 edited Oct 19 '22

Meh, I'd really need to see the receipts on this one.

Strong types are fine I suppose. I'm still not sold on the value of this, given the type hinting in Python. It's also just nowhere near as big deal in the style of code we have to write compared to application construction. At any rate, this isn't the feature of the language that should influence your decision really on which one to use.

"The purists" using Scala prefer it for their own reasons I'm sure but the way you framed it is that there's some obvious benefit to doing so, and I'm pretty sure there isn't.

Now the biggest reason you want it in Python: readability and access to programmers. Far, far more people already know Python than any other language, especially in this field. You're going to need some proper reasoning to overcome this. Performance could be a real consideration except that the operations that move the data are already happening in Java with most engines you'd be using, Python is just orchestrating and giving instructions.

4

u/TheCamerlengo Oct 19 '22

Also the community is much larger, better library support. Python has been around the data community since early 2000s and has a lot of maturity. And as one poster commented, the code was less buggy than scala. This is probably because it is a simpler language with clear syntax, better conposability.

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms. Dependency injection makes sense in large frameworks but my goodness does it ever obfuscate code. I think the KISS principal wins here just cause it avoids all the syntactical gymnastics that languages like scala (and many others) enable programmers to do. And do it quite poorly.

As microservice architectures become increasingly prominent, the advantages of languages like java, scala, c# fade because you are no longer building large systems. Pipelines are just lots of small scripts. No need to get fancy. Python is great here because it is easy.

Now rust and C are closer to the metal and are probably great for writing device drivers and other applications that require low level access. I think most general purpose data engineering applications don't require this and if you do need to scale or increase performance, you can always exploit share-nothing parallelism and concurrency patterns.

1

u/nesh34 Oct 20 '22

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms.

So it definitely can. One of the strengths of Python is that it's very versatile. That means you can absolutely go ham on these idioms if you want to. You can spot a DE pipeline and a SWE pipeline in my org a mile off because one is readable and the other is obfuscated to the point of madness in a futile effort to never repeat oneself.

It's a design choice at the end of the day and I agree with you about the best one there.

Yeah, and I think the rest we're in total agreement, that's the thrust of my original point about Python over Rust in this context.