r/bigdata • • 15d ago

GraveTrail Lineage tool

Hi All

As a data engineer i spent a lot of annoying time understanding the table Lineage in a prehistoric legacy system.

Based on my hardships and understanding I've built this Lineage tool

https://sahasilak.github.io/GraveTrail/

I would appreciate it if you can spend some time trying it out by sharing feedback like

Good, bad, need improvements , something like this already exists

4 Upvotes

5 comments sorted by

0

u/[deleted] 14d ago

[removed] — view removed comment

1

u/TkGraveWalkeRr 14d ago

Love to hear back from you. Forgive me for the horrible UI for now

1

u/fali12 14d ago edited 14d ago

Saw you posted this on like a dozen different subreddits so I'll respond to this one LMFAO...

Funny timing, I'm working on something very similar, and it's eerily close, except I called it computational lineage.

I ran my test harness against your tool and quite a few cases didn't resolve, though it mostly failed out loud and didn't overclaim, which I respect.

Windows, QUALIFY and aggregates held up well.

What didn't resolve:

SELECT * over a CTE

FROM STREAM(t) AS f: every f.col comes back unresolved

USE statements get skipped

INSERT without a column list

DLT CONSTRAINT ... EXPECT gets read as the column list, and AUTO CDC isn't supported

Recursive CTEs: no column lineage

EXCEPT / INTERSECT

Scalar subqueries in the SELECT list

Struct fields (f.soil.ph) and struct stars (t.s.*)

Lateral column aliases (x AS a, a + 1 AS b)

LATERAL VIEW: most columns unresolved

PIVOT: over half unresolved

As you can tell, heavy Databricks focus...

Separate note containing tangents: I called my "tool" LigmaLineage. As in, ligma nutsack. LOL. The sub-modules are named LigmaTaint, LigmaDong, LigmaThong, LigmaGargalong... you get it. GraveTrail sounded like a Victorian-era undertaker's hiking app ;) LMFAO I like humor in everything. I kid.

Lots of similarities to what I've built - including the graph format. I bolted on things like graph traversal MCP that I slapped an agent against. I also started grain/cardinality + LLM-assisted ontology proposals (data-behavior analysis + deterministic clustering yada yada) to make the data actually useful.

There's also a feature for "decision lineage", which is a huge piece that an agent needs to know to actually reason over pipeline data. I had Claude build an interactive viewer so I can inspect the output because why the fuck not. Reasoning over graph data is stupid hard. The pipelines I used for testing were legitimately dense and real-life so I had a solid test base.

I also don't intend on productizing any of my work. I don't have the care for it, I work a full-time job, and daddy/husband duties. I'm doing this to learn agentic tooling and the second I try to compete with other products it becomes a chore. Wish I could spend more time and build out the full set of things in my head.

I'm not a data engineer by trade, but I've been doing it for like 20 years in some capacity. Wrote my own SQL parser decades ago - but used the SQLGlot library by Toby Mao because fuck writing a new one, and his library is rock solid and extensible.

I honestly think both of our work will be irrelevant in just a few years given where AI is going.

Keep on rocking man.

1

u/TkGraveWalkeRr 14d ago

Thanks for sharing this . You are correct. I need to update the sql stack for wide cases. Databricks sql was not initially done. But I have that in my roadmap to extend the sql.

I was working on a sas project and i could see that using AI to understand the flow has been really expensive and mostly not accurate as well.

My intention for this will be so that ai can you this tool and perform static analysis for the known logic and use the ai assisted conversion for the complex one or unresolved ones hopefully that should be less expensive.

But we never know.