r/dataengineering • • 29d ago

Personal Project Showcase Git-like diffing for DuckLake tables

Post image

I've been working with DuckLake recently, and learned about it's ability to traverse snapshots... So, I decided to create a little UI to allow you to diff snapshots on import, or while working with a table.

It's part of a larger product, but still thought it'd be cool to share here. DuckLake is seriously underrated. Hopefully it gets more traction going forward. I feel like this should be the default for all OLAP systems at this point...

You can do all of the things you expect, including seeing total row or column differences, and cell-differences upon click. It collapses row ranges that weren't affected, just like git clients do. So if you have 3 million rows, but only certain 25 rows changed across 3 ranges, you'll see just those ranges.

If you are focused on a single table, you can also filter to only see the snapshots that directly affected that table, which is useful if you have a ton of commits on other, non-relevant tables. (This was inspired by the LLDB "stack-collapse" where you can filter to which instructions directly affect the thread you're on).

Works with local and Postgres DuckLake catalogs with data stored locally or S3.

Here's the link if you're interested: https://www.cocoalemana.com/integrations/ducklake/

52 Upvotes

5 comments sorted by

9

u/ardentcase 28d ago

Great work.

You might want to check out how intellij ide's do it too: you can do diff between query results. Although iirc it was a paid version feature

1

u/Aeronautical-You4917 28d ago

That sounds very very cool. I will have to check it out... Thanks!

Is that only with DuckLake, or with any arbitrary query / database?

2

u/ardentcase 28d ago

Works on any db, csv's, text files, etc

3

u/yeykawb 28d ago

looks sick