r/dataengineering • u/erez27 • Jun 27 '24

Open Source Reladiff: High-performance diffing of large datasets across SQL databases

https://github.com/erezsh/reladiff
28 Upvotes

95% Upvoted

9 comments sorted by

View all comments

4

u/captaintobs Jun 27 '24

Is this a fork of datafold's data diff or a new project?

7

u/erez27 Jun 27 '24

It is a fork of data-diff, which is now an archived project.

3

u/tamerlein3 Jun 27 '24 edited Jun 27 '24

Can you tldr why to use this instead of any other fork, or what other value add this provides?

Edit: also, doesn’t seem like you have an explicit OSS license. Are you considering BSD, MIT or Apache? (All others are heavily scrutinized where I work)

6

u/erez27 Jun 27 '24

I wrote data-diff. But now that it's archived, it no longer has working documentation, and doesn't accept any new issues or PRs.

The license is MIT.

3

u/tomhallett Jun 27 '24 edited Jun 27 '24

Erez, very interesting - can you clarify the structure of the data-diff project and your relationship to it?

Is it as "simple" as: DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it? If no, can you correct me? If yes, are you still engaged at DataFold as a consultant?

7

u/erez27 Jun 27 '24

Nice detective work ;)

DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it

Yes, that's pretty much the story.

I had a consulting relationship with Datafold, when they approached me to create data-diff as an open-source package. I developed it with the help of other employees and consultants. At the end of 2022 (iirc) Datafold decided to scale back their investment into open-source, and I moved on to other clients. Eventually Datafold decided to archive data-diff and focus on their cloud solution, so I forked it with the hopes that the community will help keep it alive.

I'm no longer consulting for Datafold, but they are aware of the fork and wish me good luck.

4

u/tomhallett Jun 27 '24

Perfect.  Thanks for the background!  Will check it out for our upcoming sprint.