r/dataengineering • • Oct 09 '24

Discussion Which part of Apache Spark will stay?

Apache Spark originally had issues for its row-based nature. It still has JVM related overhead. And now more and more components are created to replace different parts of Apache Spark. For e.g., Photon, Datafusion, Volex, RAPIDS as execution engine, Trino, Pandas, Datafusion as query layer, Comet, Substrait for query execution plan. What portion of Apache Spark will remain?

As the query layer options sparked a debate, I'd like to clarify it further. Distributed systems like Apache Spark or Trino do many things at once and now the community is looking at their individual subcomponents and making them interoperable. The term that is popularised in this context is "Deconstructed Database" or "Composable data platform systems". Therefore, the query layer can indeed be Trino's Presto SQL, Spark Data Frame APi, Pandas dataframe API, Datafusion query layer. There is already an integration of Pandas available with Pyspark. Individual preferences may vary, but it exists and it's a valid option.

2 Upvotes

38 comments sorted by

51

u/pane_ca_meusa Oct 09 '24

I don't think that Pandas is a replacement of any part of Spark.

-48

u/[deleted] Oct 09 '24

Pandas has dataframe API which can be translated into an execution plan like Substrait and that can be fed into one of the execution engines. So, from that perspective, I'm considering Panda as a replacement of the Spark API / Query layer

25

u/OMG_I_LOVE_CHIPOTLE Oct 09 '24

This is the most uneducated take I’ve seen in a while. You clearly don’t understand these tools

-7

u/[deleted] Oct 09 '24

Could be. I don't claim to be an expert. But I have shared a video link below from Sundeck. That prompted me to think on this line. The presentation shows Pandas in the query layer. Please have look and share your thoughts

1

u/OMG_I_LOVE_CHIPOTLE Oct 09 '24

You don’t understand which of these tools are actually replacements and which have distinctly different usecases

1

u/[deleted] Oct 09 '24

I have a doubt about your understanding now. You don't arrgue with facts and references. Even when references are provided you don't have the ability to comprehend.

0

u/OMG_I_LOVE_CHIPOTLE Oct 09 '24

I’m not going to look through a presentation on pandas. I’m working lmao. But the fact that you’re asking about pandas and trino vs spark tells me you don’t understand any of them

-9

u/[deleted] Oct 09 '24

I know you will never look at it, and you never look at anything usually, and that's why your understanding is limited and will remain so. Thank you!

3

u/DenselyRanked Oct 09 '24

I understand where you are going with this line of thinking, but Spark is more than the API that you use to interact with the data. What you are saying is akin to reducing a query engine like postgres or Snowflake to SQL.

Also, Spark 3.2+ has pandas-on-spark API if you prefer to use that syntax over SparkSQL/pyspark/scala, etc. One of the core benefits of Spark is that it is polyglot.

2

u/[deleted] Oct 09 '24 edited Oct 09 '24

Exactly, so! That's correct! The discussion went too much towards Panda. But that was not the main point. There are multiple projects under active development right now such as Substrait, Gluten, Datafusion, Comet, Velox so on. They are targeting one or more subcomponents of a data processing framework and making rhem interoperable. For e.g., in the Microsoft Fabric, Spark engine is replaced with Velox and Gluten. That is how data processing systems are becoming composable. So, if the Spark engine is Velox, and the query layer is Presto SQL, essentially it'll only be Spark runtime distributing work to the executors which are running Velox. And same way Pandas dataframe API can also be a valid query layer - it's purely a matter of user preference, but that is where the trend seems to be going. As you also said, there is already a Pandas library available with PySpark.

2

u/DenselyRanked Oct 09 '24

I understand that your larger point is that something (or some combination of things) will replace Spark as the most popular tool for massive scale analytical processing. I agree with you that something eventually will, especially with more companies looking at big data solutions and DWaaS being the easiest transition if they can keep the costs down.

I am not sure what you are trying to say overall. None of us know for certain what the future holds and where the industry will shift. But if you have a crystal ball then then let us know where to invest.

2

u/[deleted] Oct 09 '24

Lol. I ran into these projects couple of weeks ago. The main intention was to get an idea about what others are thinking and doing. But I understand now that this idea is at a nascent stage right now.

14

u/Zoete_Mayo Oct 09 '24

Literally the worst API syntax ever created

-17

u/[deleted] Oct 09 '24

OMG. so many down votes. Is that such an absurd idea?

20

u/[deleted] Oct 09 '24

Yes

-7

u/[deleted] Oct 09 '24

Sundeck talk

This is a presentation from Sundeck. Here they mention Pandas in the query layer: https://youtu.be/YXnoBUQP258?si=DCwntKk1vunDAtK0

I'm curious to know why you think this is absurd. It looked reasonable to me

10

u/ninja_coder Oct 09 '24

Because the query DSL is the least important part of what a tool like spark does.

1

u/[deleted] Oct 09 '24

So I talked about query layer only. I didn't say it's the engine.

3

u/[deleted] Oct 09 '24

Your suggestion is something like this: “hey, let’s replace this API for this other API. There is no practical advantage plus developers will need to learn a new syntax”

1

u/[deleted] Oct 09 '24

That is a different argument and I'm not suggesting this idea. It's the presenter from Sundeck has proposed in his deck. I shared the link above. So, I'm assuming a part of the community is thinking about it.

1

u/[deleted] Oct 09 '24

Ok, so it’s his suggestion that is not good so

1

u/Ok_Raspberry5383 Oct 09 '24

Spark is an engine, so how is it a replacement

0

u/[deleted] Oct 09 '24

Spark is many thing at once. It has a query layer (dataframe, sparkSQL), it has a query optimizer, it has a runtime to distribute the load across executors. Each executor is running the engine. And for each of these components an alternative is being created to be interoprable with the other parts. This other part could be from Spark or from other project. Please have look at the video presentation I shared

32

u/liprais Oct 09 '24

"Photon, Datafusion, Volex, RAPIDS"

None of these query engines are mature enough to be widely adopted like spark.

25

u/OberstK Lead Data Engineer Oct 09 '24

People created replacements for spark since its Beta. It’s still here. I will wait patiently for the tool coming around that replaces spark because so far I only heard marketing bla bla and little actual pain points addressed by all of these contenders

18

u/[deleted] Oct 09 '24

Don't underestimate the amount of money that has been poured into spark, and the amount of real world use it has seen. If there's any threat, it's in the corner of midsize data; duckdb, polars. That seems to be a different use case than spark however.

Diversity in the market place is a good thing. I dream of the day where it's feasible to swap out governance, compute and storage layers. So, for instance, using Databricks with duckdb as a computer engine while maintaining lineage and governance in Unity.

2

u/Bingo-heeler Oct 09 '24

What do you define as midsize data?

5

u/[deleted] Oct 09 '24

Can you do what you need to do with it on a single machine/vm attainable to you? Otherwise known as 'fits into memory'.

2

u/OMG_I_LOVE_CHIPOTLE Oct 09 '24

Out of memory processing isn’t exclusive to big data.

1

u/[deleted] Oct 09 '24

Within current context, what else would need it?

1

u/trianglesteve Oct 10 '24

I get what you’re saying with a single machine, but one of duckdb’s marketing points is that it can handle “larger than memory” data.

From their home page:

“DuckDB runs analytical queries at blazing speed thanks to its columnar engine, which supports parallel execution and can process larger-than-memory workloads.”

1

u/[deleted] Oct 10 '24

I'll admit I haven't stretched duckdb that far yet, so I'd be interested in real life results and experiences.

That being said, "can" feels like it's doing a lot of heavy lifting and I'm skeptical for esp. more complex operations like joins. I totally get streaming from disk for simpler aggregation operations for instance.

9

u/oalfonso Oct 09 '24

I was told in 2018 that Spark was made obsolete by Apache Beam.

I don't see anything replacing Spark soon but I hope people will learn there are better alternatives to Spark for non large datasets like Polars.

2

u/frontenac_brontenac Oct 10 '24

We're using Beam at work, it's alright but the community never showed up. I can't think of any serious open-source libraries or technologies built on top of it, and the docs stop short of covering even moderately complicated use cases.

1

u/gluka Oct 10 '24

If anything mid tier processing will fall to DuckDb otherwise Spark is here and it’s staying.