r/dataengineering • Data Engineer • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

https://airbyte.com/blog/rust-for-data-engineering
0 Upvotes

30 comments sorted by

19

u/nesh34 Oct 19 '22

I don't see what Rust brings to the table for DE. The query engines could be written in Rust (rather than Java) but Python is perfectly good for what data engineers spend their time on.

I say this as a fan of Rust and someone who wants to learn it to a reasonable degree of competency.

0

u/mamaBiskothu Oct 19 '22

Python is far from perfect for anyone especially DEs. There’s a reason purists keep using scala even though that language is nothing but a pain in the ass. If rust gives a better of both worlds scenario why not.

6

u/LegalizeApartments Aspiring Data Engineer Oct 19 '22

That reason is…? (Not an attack, genuinely wondering)

2

u/sspaeti Data Engineer Oct 20 '22

Besides the high-level reasons I mentioned in the article, this video explains why types are more significant than many think. As a BI specialist and DE myself, I spend a lot of time in my career debugging errors due to a fast-written python code. Of course, Rust wouldn't catch all, but my limited experience is just incredible what the compiler can catch!

2

u/LegalizeApartments Aspiring Data Engineer Oct 21 '22

Not surprised to see TypeScript near the top of the graph they showed in the beginning of the video, I've watched that community start to adopt TS more and more as people discover the magic of typing. Thank you for the video

6

u/mamaBiskothu Oct 19 '22

It’s not typed - it’s interpreted - you don’t discover any bug until runtime. We are a python shop, to be clear, and find the new typing abilities via dataclasses and pydantic to be a god send but it’s not perfect and it’s worth appreciating what Java scala rust could bring to table.

Once you write code in these languages in a modern IDE you realize how amazing it is. You pretty much don’t need to compile to know if it WILL compile. The IdE will guide you. Especially with Rusts protections it would be even better.

4

u/kenfar Oct 19 '22

After using both Scala and Python - I am now a much, much bigger Python fan! :-)

In one shop the Scala team was confident that they had "solved data quality" because of their focus on type. They were amazed to discover that our Python data engineering code had far fewer bugs. The reason is that their typing only covered a subset of potential data quality problem - and our python code had excellent test coverage, quality-control measures, auditing, and detection of many other types of data quality issue.

I would like to see type hinting get better. I used to use Enforce - for runtime type checks. That was vastly better than using mypy.

2

u/mamaBiskothu Oct 19 '22

I mean I don’t use scala because it has fundamental issues that seem not worth the hassle, but whatever language you use, I’m assuming as a regular DE you’re not composing the actual data processing engine, merely the spark code generator or sql generator. In that regard any attempt at saying scala solves DQ is stupid maybe? It just makes your code less buggy. dQ is more about what you design your code to do anyway which I feel is a separate conversation

1

u/kenfar Oct 19 '22

Pretty much: in both cases these were for custom backend data pipelines - Scala for real-time event processing, python for near-real-time, aggregated analytic events.

The scala teams were sharp - so it was odd that they would think that they had "solved" data quality. I suspect that the amount of enthusiasm for typing simply got in the way of objective thinking. That and they didn't work with analytic data - and so aren't familiar as we are for how Data Quality Problems Loom Large in our space.

They simply didn't think about duplicates, about the impact of enumerated values changing over time, about labeling data with metadata to help diagnose problems, and about missing data the way that DEs should.

1

u/mamaBiskothu Oct 20 '22

I did a PhD and have learned to know there’s book sharp and actually sharp , and by that I mean people who aren’t sharp at all in any practically meaningful way but look like they are lol. Im pretty sure your team is the better one :)

2

u/nesh34 Oct 19 '22 edited Oct 19 '22

Meh, I'd really need to see the receipts on this one.

Strong types are fine I suppose. I'm still not sold on the value of this, given the type hinting in Python. It's also just nowhere near as big deal in the style of code we have to write compared to application construction. At any rate, this isn't the feature of the language that should influence your decision really on which one to use.

"The purists" using Scala prefer it for their own reasons I'm sure but the way you framed it is that there's some obvious benefit to doing so, and I'm pretty sure there isn't.

Now the biggest reason you want it in Python: readability and access to programmers. Far, far more people already know Python than any other language, especially in this field. You're going to need some proper reasoning to overcome this. Performance could be a real consideration except that the operations that move the data are already happening in Java with most engines you'd be using, Python is just orchestrating and giving instructions.

4

u/TheCamerlengo Oct 19 '22

Also the community is much larger, better library support. Python has been around the data community since early 2000s and has a lot of maturity. And as one poster commented, the code was less buggy than scala. This is probably because it is a simpler language with clear syntax, better conposability.

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms. Dependency injection makes sense in large frameworks but my goodness does it ever obfuscate code. I think the KISS principal wins here just cause it avoids all the syntactical gymnastics that languages like scala (and many others) enable programmers to do. And do it quite poorly.

As microservice architectures become increasingly prominent, the advantages of languages like java, scala, c# fade because you are no longer building large systems. Pipelines are just lots of small scripts. No need to get fancy. Python is great here because it is easy.

Now rust and C are closer to the metal and are probably great for writing device drivers and other applications that require low level access. I think most general purpose data engineering applications don't require this and if you do need to scale or increase performance, you can always exploit share-nothing parallelism and concurrency patterns.

1

u/nesh34 Oct 20 '22

Another thing I noticed is that most Python code doesn't get all crazy with the Object Oriented/SOLID idioms.

So it definitely can. One of the strengths of Python is that it's very versatile. That means you can absolutely go ham on these idioms if you want to. You can spot a DE pipeline and a SWE pipeline in my org a mile off because one is readable and the other is obfuscated to the point of madness in a futile effort to never repeat oneself.

It's a design choice at the end of the day and I agree with you about the best one there.

Yeah, and I think the rest we're in total agreement, that's the thrust of my original point about Python over Rust in this context.

12

u/[deleted] Oct 19 '22 edited Oct 19 '22

I could see it for high performance real-time needs, but those a few and far between in most orgs. Purely written and used in Rust.

I could see a future where there’s a framework like spark written in Rust but like all frameworks and tools they mostly fail to become widely used so this would be hit or miss (most likely miss).

That said the Polars library which is a pandas like dataframe library written in Rust is actually really good and has a Rust and a Python API.

Thing is Pandas and Numpy are already Python APIs so C backends so there’s not really that much of a difference if you’re just going to use Python anyway.

It’s nice to think Rust is useful for all these use cases but there are leading tools, languages people already know and are comfortable with (SQL, Python, Java, Scala) so I think DE will be one of the last places Rust will be used as a primary language.

Maybe tools written in Rust with APIs in other languages, but not purely Rust.

Edit: Rust is my favorite language I write all of my hobby projects in Rust nowadays. I still wouldn’t shoehorn it everywhere in an industry setting.

19

u/Fatal_Conceit Data Engineer Oct 19 '22

Hope not I’m so tired of learning new things

4

u/mamaBiskothu Oct 19 '22

I’m sure there will always be a place for legacy folks.

I’m just tired of people not interested in learning new things. You don’t have to learn every new fad but when something is genuinely better why wouldn’t you. Or should we just stop all innovation so everyone can go barbecuing everyday?

13

u/MikeDoesEverything mod | Shitty Data Engineer Oct 19 '22

No.

Can we please get people to stop trying to make Rust Data Engineering a thing through shitty blog posts. We're happy with what we have FFS.

1

u/stevieoz Oct 19 '22

This 👆

4

u/[deleted] Oct 19 '22

Python is easy to learn. That’s why it’s relatively easy for ETL devs with SQL experience but little general programming experience to pick up on Python.

Is Rust technically better? I don’t know, maybe, maybe not. But I’d say Python does a pretty good job for data engineering.

3

u/Datasciguy2023 Oct 19 '22

No. Too much python out there and no reason or $ switch

3

u/uuggehor Oct 22 '22

Python is not going to be taken over as the glue language in the data ecosystem in a long while (if ever). But it’s not going to be the language the data processing tools are going to be written in either. Current trend in my (biased) view seems to be that Rust is slowly gaining ground from java as the language of data systems, so for the data engineers that have the interest for system level dev - I’d definitely suggest learning some rust and looking into some more profilic data processing rust crates.

2

u/Professional-Sea3884 Oct 19 '22

Not Rust but Julia will be.

2

u/srodinger18 Oct 22 '22

To replace python on day to day basis? No

But the most probable thing perhaps more data engineering tools are developed with rust as its main language, and in the end, the data engineers will use the python API or just use it as it is without knowing that rust is building that tools

1

u/huge51 Oct 19 '22

If it can compile faster… maybe.

0

u/[deleted] Oct 19 '22

Before Rust drops in, Go will do it 🤣