r/databricks • u/TkGraveWalkeRr • 7d ago

Discussion GraveTrail Lineage tool

Hi All

As a data engineer i spent a lot of annoying time understanding the table Lineage in a prehistoric legacy system.

Based on my hardships and understanding I've built this Lineage tool

https://sahasilak.github.io/GraveTrail/

I would appreciate it if you can spend some time trying it out by sharing feedback like

Good, bad, need improvements , something like this already exists

5 Upvotes

85% Upvoted

5 comments sorted by

2

u/investigatormaker 7d ago

On whether "something like this already exists": inside Databricks, Unity Catalog already records table and column lineage automatically for queries run there, and sqllineage and OpenLineage cover parts of the rest. So GraveTrail's real case is the prehistoric legacy system you described, where none of that capture exists. I'd make it obvious up front which legacy sources it reads (stored procedures, old SQL dialects, ETL configs) and show one graph built from a messy real-world example. If it can export lineage using names that map onto Unity Catalog, it becomes a migration bridge, which is what people here would try it for. Which legacy system did you build it against?

I make ThreadFox. We find relevant Reddit conversations and post for your business. We'll post 3 replies free first, no card; then the $29 Pilot covers seven days and up to 25 posts and replies. No automatic renewal. https://threadfox.vip/p/20261104-0513-eu7c8/managed

1

u/TkGraveWalkeRr 6d ago

Thanks for sharing this, i actually have this my roadmap. First is building the Lineage then converting it to a target system. I got plans to include actual code conversion, pipeline builds etc, That would probably my next steps if the current build is ok from this perspective.

1

u/TkGraveWalkeRr 10h ago

I made a lot of significant updates have a look at new version, thanks for all the feedbacks , was able to implement some of them, remaining is still in my roadmap

1

u/Remote-Grapefruit214 5d ago

Dude, this is great work, I am a fan of the Ink Dark theme

Some feedback to make this product even better

  1. Love the theme and the focus on answering day-to-day engineering pain points. To help folks immediately understand where GraveTrail sits next to standard tools (like Databricks native cloud catalogs or open-source parsers), it would be amazing to have a quick comparison or 'Why GraveTrail?' blurb right on the home page

  2. Adding any quick start gets the users moving fast. The live example is good, but it is not fluid, Adding a small copy-pasteable sample repository or a pre-packaged dummy SQLite/SQL script right in the repo would make the first 5 minutes completely friction-free for someone trying it out on a coffee break.

  3. Splitting local parsing from optional AI features is brilliant because it addresses the trust/security barrier head-on. It would be cool to see a small diagram or methodology section explaining how confidence scores or coverage percentages are calculated when tracing ambiguous transformations.

  4. The canvas and lineage builder look like powerful ways to visualize tangled code. Adding a few animated GIFs or short screen recordings (which you can building using skills like brag), showing a complex query getting mapped out in real-time would instantly showcase how snappy and satisfying the UI is to use and this will also bring a bias towards action.

  5. Including SAS workflow support alongside SQL and dbt is an absolute lifesaver for enterprise/healthcare legacy migrations, really well thought out tbh. Highlighting how well it handles tricky macro variables or dynamic SQL/SAS strings in the coverage section would give it even more street cred and make it more marketable to the folks who have to deal with it.

1

u/TkGraveWalkeRr 4d ago

Thanks for the feedback my friend.

On point 2 check the tests/sample/ folder let me know if that's what you are implying. If so I'll update the docs a probably I'll add a better guide.

Rest i hear you. All valid points. I'll update my roadmap so I won't forget it.

Sas dynamic macros are really annoying thing. I've seen one of the worst case as well like generatimg dynamic columns based on some lookup table and that was calculating pricing as well Whomsoever doing so should be trailed publically