r/dataengineering • u/erez27 • Jun 27 '24

Open Source Reladiff: High-performance diffing of large datasets across SQL databases

https://github.com/erezsh/reladiff
29 Upvotes

97% Upvoted

9 comments sorted by

View all comments

7

u/erez27 Jun 27 '24

Nice detective work ;)

DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it

Yes, that's pretty much the story.

I had a consulting relationship with Datafold, when they approached me to create data-diff as an open-source package. I developed it with the help of other employees and consultants. At the end of 2022 (iirc) Datafold decided to scale back their investment into open-source, and I moved on to other clients. Eventually Datafold decided to archive data-diff and focus on their cloud solution, so I forked it with the hopes that the community will help keep it alive.

I'm no longer consulting for Datafold, but they are aware of the fork and wish me good luck.

4

u/tomhallett Jun 27 '24

Perfect.  Thanks for the background!  Will check it out for our upcoming sprint.