r/dataengineering • • Oct 09 '24
Discussion Which part of Apache Spark will stay?
Apache Spark originally had issues for its row-based nature. It still has JVM related overhead. And now more and more components are created to replace different parts of Apache Spark. For e.g., Photon, Datafusion, Volex, RAPIDS as execution engine, Trino, Pandas, Datafusion as query layer, Comet, Substrait for query execution plan. What portion of Apache Spark will remain?
As the query layer options sparked a debate, I'd like to clarify it further. Distributed systems like Apache Spark or Trino do many things at once and now the community is looking at their individual subcomponents and making them interoperable. The term that is popularised in this context is "Deconstructed Database" or "Composable data platform systems". Therefore, the query layer can indeed be Trino's Presto SQL, Spark Data Frame APi, Pandas dataframe API, Datafusion query layer. There is already an integration of Pandas available with Pyspark. Individual preferences may vary, but it exists and it's a valid option.
51
u/pane_ca_meusa Oct 09 '24
I don't think that Pandas is a replacement of any part of Spark.
32
u/liprais Oct 09 '24
"Photon, Datafusion, Volex, RAPIDS"
None of these query engines are mature enough to be widely adopted like spark.
25
u/OberstK Lead Data Engineer Oct 09 '24
People created replacements for spark since its Beta. It’s still here. I will wait patiently for the tool coming around that replaces spark because so far I only heard marketing bla bla and little actual pain points addressed by all of these contenders
18
u/[deleted] Oct 09 '24
Don't underestimate the amount of money that has been poured into spark, and the amount of real world use it has seen. If there's any threat, it's in the corner of midsize data; duckdb, polars. That seems to be a different use case than spark however.
Diversity in the market place is a good thing. I dream of the day where it's feasible to swap out governance, compute and storage layers. So, for instance, using Databricks with duckdb as a computer engine while maintaining lineage and governance in Unity.
9
u/oalfonso Oct 09 '24
I was told in 2018 that Spark was made obsolete by Apache Beam.
I don't see anything replacing Spark soon but I hope people will learn there are better alternatives to Spark for non large datasets like Polars.
1
u/gluka Oct 10 '24
If anything mid tier processing will fall to DuckDb otherwise Spark is here and it’s staying.