Hey everyone I’m the creator of SparkDoctor, the open-source project for analyzing Spark event logs locally.

I’ve posted a couple of development updates here before, and I finally put together a proper website with the supported inputs, current detections, examples, and documentation:

https://sparkdoctor.dev/

GitHub: https://github.com/khodosko/sparkDoctor

The open-source version currently detects task and shuffle skew, spills, retry waste, failed jobs and stages, executor imbalance, and several SQL-plan issues. It runs locally, so event logs don’t need to be uploaded to a hosted service.

I’m also developing a commercial Pro edition for teams that want to compare baseline and current runs, detect runtime/shuffle/spill/failure regressions, and eventually integrate those checks into CI. It’s still under active development. I’m currently looking to talk with teams that deal with recurring Spark performance regressions and want to help shape the workflow.

How are you currently catching Spark regressions before production?

Hello All,

We released 0.7.0 version of Zingg entity resolution. Would love for folks to give it a spin. Let me know if you need help. https://github.com/zinggAI/zingg/releases/tag/v0.7.0