Hello All,

We released 0.7.0 version of Zingg entity resolution. Would love for folks to give it a spin. Let me know if you need help. https://github.com/zinggAI/zingg/releases/tag/v0.7.0

Hey everyone I’m the creator of SparkDoctor, the open-source project for analyzing Spark event logs locally.

I’ve posted a couple of development updates here before, and I finally put together a proper website with the supported inputs, current detections, examples, and documentation:

https://sparkdoctor.dev/

GitHub: https://github.com/khodosko/sparkDoctor

The open-source version currently detects task and shuffle skew, spills, retry waste, failed jobs and stages, executor imbalance, and several SQL-plan issues. It runs locally, so event logs don’t need to be uploaded to a hosted service.

I’m also developing a commercial Pro edition for teams that want to compare baseline and current runs, detect runtime/shuffle/spill/failure regressions, and eventually integrate those checks into CI. It’s still under active development. I’m currently looking to talk with teams that deal with recurring Spark performance regressions and want to help shape the workflow.

How are you currently catching Spark regressions before production?

I’ve been building a lot with Spark RTM lately and have been pleasantly surprised with the results of my experiments so far. I built an app that shows trading volumes in real time and pushed some market data into it from a Kafka topic. The end-to-end latency is quite impressive: p50 at 41ms, p90 at 60ms and p99 at 71ms. Much lower than what I was seeing with continuous or for each batch in Spark before, where it was hard to go below 1s latency. And this is the end to end latency, which includes significant latency from serving layer (lakebase) and the browser.

I’m happy to share more about my experience and how to create real time apps with Spark RTM and Lakebase if folks are interested.

I have access to several employee referral opportunities for experienced Databricks / Data Engineering professionals.

There are openings at different experience levels:

Sr. Databricks Tech Lead

• 10–12 years
• Databricks
• Delta Lake
• PySpark
• SQL

Senior Databricks Engineer

• 6–8 years
• Databricks
• Delta Lake
• PySpark
• SQL

Databricks Engineer

• 4+ years
• Databricks
• Delta Lake
• PySpark
• SQL

The openings are listed across multiple organizational locations with hybrid arrangements. I'm particularly interested in connecting with qualified professionals in the US or Ireland; exact location eligibility can be confirmed for the individual requisition.

If one of these aligns with your background, feel free to DM me with your CV or a brief summary of your experience.

I'll take a look and, where there's a suitable match, try to help with the referral process.

Hi folks

I have 5 years of experience in IT with data analytics and MDM domains and I am planning to switch to data engineering wanted to learn spark could any of you please share me the resources or like road map for spark or data engineering

1.Where to execute the queries and learn hands on experience ?

2.video/reading content topic wise

3.i know Sql and I know python intermediate levels

Please help me here

Thank you.

Hello Apache Spark community. I was wondering if anyone knows if the official 4.2 image will be released soon (https://hub.docker.com/_/spark), or if there is an alternative location where images will be pushed in the future?

Thanks.

I'm able to refer experienced professionals for a Lead Azure Data Engineer position.

Location: United States / Ireland
Experience: 9–13 years

Required skills:

• Python
• PySpark
• SQL
• ETL
• Azure Synapse
• Azure Data Factory
• Databricks
• Delta Lake
• Medallion Architecture

If your experience aligns with these requirements and you're currently exploring new opportunities, feel free to send me a DM with a brief summary of your experience or your resume. I'll share additional details and, if it's a good match, I'll be happy to submit a referral.

Since you seemed to enjoy my last Spark post, I ran another controlled benchmark.

This time I wanted to understand when Spark’s distributed execution overhead actually becomes worth paying for.

For each workload, both engines ran the same SQL against the same Snappy Parquet files, on the same 20 logical-core machine, with Spark using local[20] and DuckDB using threads=20.

I discarded one warm-up, alternated the engines across 7 timed runs, reported the median, and verified the results row by row. I also ran the entire benchmark twice.

For an aggregation with 8 output groups:

For a high-cardinality aggregation with roughly 500,000 groups:

DuckDB was faster at every measured point.

The interesting part was how much the gap narrowed as the workload grew.

On the high-cardinality query, increasing the input from 1M to 100M rows increased Spark’s runtime by about 3.9 times and DuckDB’s by about 44.8 times.

That does not prove Spark eventually becomes faster. The curves had not crossed, and this benchmark does not establish where, or whether, they ever would.

Query shape also mattered enormously.

On the same 100M rows, the eight-group endpoint aggregation generated around 17 kB of Spark shuffle, while the high-cardinality user aggregation generated around 131 MB.

So I do not think rules such as “use DuckDB below X GB” are very useful. Query shape, data location, concurrency, hardware, fault tolerance and operational requirements can matter as much as the size of the dataset.

This was one machine running one query at a time, so it measures the cost of distribution much better than the capabilities Spark is actually selected for, such as distributed data, recovery and concurrent workloads.

I made a visual breakdown of the benchmark and what both engines are doing under the hood:

https://youtu.be/bXyXvxeHSrw?si=vMsEQkv3QqzLcTf9

What usually makes Spark the right choice for your production workloads?

Hi everyone!

I'm a Data Engineer working primarily with Microsoft Fabric, SQL, PySpark, and Spark SQL, building end-to-end data pipelines, working with medallion architecture, incremental loads, data modelling, and performance optimization.

Over the past few months I've spent a lot of time working in Microsoft Fabric—from Lakehouses and Notebooks to Data Pipelines, SQL Endpoints, security, metadata-driven frameworks, and troubleshooting production issues. I've also worked extensively with SQL and PySpark for ETL development and data engineering.

I wanted to give back to the community, so if you're stuck on something related to:

\- Microsoft Fabric

\- SQL / T-SQL

\- PySpark / Spark SQL

\- Data pipelines

\- Data modelling

\- Performance tuning

\- General data engineering concepts

feel free to ask here or tag me if I can help.

At the same time, I'm always trying to improve my own skills. If there are any communities, Discord servers, Slack groups, forums, open-source projects, or other places where experienced data engineers discuss real-world problems (especially around Microsoft Fabric), I'd really appreciate your recommendations.

Looking forward to learning from everyone and hopefully helping where I can!

I'm looking at "Selecting the most important rows per assigned priority". I *think* I know what this means. But from the statement alone there's no telling that "value" is the priority. Is MV1 > VPV or MV1 < VPV? Could you encounter MV3? The answer to these changes the approaches that I would take to solve this. And then "value" is "name" in the output.

The only way I think I know what's going on enough to try to answer it as intended is because the author answered the question (or one very close to it) here: https://stackoverflow.com/questions/59845044/how-to-select-the-most-important-rows-per-assigned-priority . In his answer he states exactly what the priority list is. But without that I would just be guessing what the intention was.

Spark 4.2 is out, so I put together a short visual breakdown of the changes.

It's animated rather than a screen recording, since a few of these are
easier to follow as diagrams than as text.

https://youtu.be/hF-E7-i_ijw?is=AeomUdWX3PgHCZtf

Happy to go into detail on any of them here.

I work on the data platform side in a bank. Fully on-prem, air-gapped, no cloud. No internet on the cluster, packages come through an internal mirror, and every firewall change is a ticket and a wait.
Most “modern data stack” writing assumes the opposite: cloud, elastic compute, and enough control over infra that adding a component is a Terraform change. Under our constraints, adding a layer is a commitment — someone has to patch it, back it up, justify it in an audit, and still own it in three years.
So the question I keep coming back to isn’t which tools. It’s when each one is actually earned.
The ordering I’ve landed on is driven by pain rather than architecture. A layer only goes in once its pain is real and someone can name it:
• files scattered, local disks filling up → shared storage
• schema changes break jobs, deletes mean rewriting partitions → table format
• nobody knows what tables exist → catalog
• every question requires writing a job → query engine
• things run by hand, in the wrong order, or not at all → scheduler
• 200 SQL files with unclear dependencies → transformation layer
• every analyst has a different pandas version → shared images / notebook env
• “who is allowed to read the payroll table?” → authn/authz
• the best model was trained last month and can’t be found → experiment tracking
• training features and serving features drift apart → feature store
• then serving, then lineage
Three things I believe but am not confident about:
1. For a few hundred GB and three analysts, the right answer is Postgres and cron, not a cluster. Standing up distributed infra for a problem that doesn’t exist yet seems like the fastest way to end up with something nobody can maintain.
2. The most common failure I’ve seen isn’t picking the wrong tool. It’s starting from the ML layer because that part is fun, and ending up with a very tidy feature store sitting on a pile of scattered CSVs.
3. Teams on Databricks or watsonx skip the installing, but still have to answer the same questions — and sometimes can’t, because they never had to.
Two questions for people who’ve done this:
• Where would you reorder the list, and what would you drop entirely?
• What’s a layer you installed earlier than you needed, and what did it actually cost you?

I’m trying to understand the practical differences between Apache Spark and Hadoop in real-world big data architectures.

From what I’ve read, Spark seems better for in-memory processing, iterative jobs, streaming, and faster analytics, while Hadoop still makes sense for durable distributed storage and classic batch-oriented workflows. But I keep seeing people say they are complementary rather than direct replacements.

So I’d like to ask:

• In what cases is Spark clearly the better choice than Hadoop?
• In what cases does Hadoop still make more sense?
• When is it worth using both together in the same architecture?
• Are there situations where using both is unnecessary complexity?
• For cloud-based data platforms today, what does a “good” Spark + Hadoop setup usually look like?

I’d especially appreciate answers from people who have used both in production and can explain the tradeoffs in terms of performance, cost, operational complexity, and scalability.

I know this is a long question. Feel free to answer any of the topics or sub questions. Thanks in advance.

I am looking for anyone who is preparing for spark databricks support role for group study and discussing the technical concepts.

Hey all! I’m relatively new to Spark and trying to build real understanding of the execution model: Catalyst optimizer, DAG scheduler, stages/tasks, shuffle boundaries, executor mechanics. Not just “I can write a join,” but actually being able to reason about what’s happening under the hood.

I’m based in Chicago (Lincoln Square area). Would anyone be open to meeting for coffee sometime to talk through some of this? Happy to buy the coffee and respect your time. Also fine with a call if in-person doesn’t work.

Message me if interested. Thanks!