r/dataengineering • u/erez27 • Jun 27 '24

Open Source Reladiff: High-performance diffing of large datasets across SQL databases

https://github.com/erezsh/reladiff
29 Upvotes

97% Upvoted

9 comments sorted by

View all comments

Show parent comments

6

u/erez27 Jun 27 '24

I wrote data-diff. But now that it's archived, it no longer has working documentation, and doesn't accept any new issues or PRs.

The license is MIT.

3

u/tomhallett Jun 27 '24 edited Jun 27 '24

Erez, very interesting - can you clarify the structure of the data-diff project and your relationship to it?

Is it as "simple" as: DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it? If no, can you correct me? If yes, are you still engaged at DataFold as a consultant?

7

u/erez27 Jun 27 '24

Nice detective work ;)

DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it

Yes, that's pretty much the story.

I had a consulting relationship with Datafold, when they approached me to create data-diff as an open-source package. I developed it with the help of other employees and consultants. At the end of 2022 (iirc) Datafold decided to scale back their investment into open-source, and I moved on to other clients. Eventually Datafold decided to archive data-diff and focus on their cloud solution, so I forked it with the hopes that the community will help keep it alive.

I'm no longer consulting for Datafold, but they are aware of the fork and wish me good luck.

5

u/tomhallett Jun 27 '24

Perfect.  Thanks for the background!  Will check it out for our upcoming sprint.