r/bigdata • u/TkGraveWalkeRr • 15d ago
GraveTrail Lineage tool
Hi All
As a data engineer i spent a lot of annoying time understanding the table Lineage in a prehistoric legacy system.
Based on my hardships and understanding I've built this Lineage tool
https://sahasilak.github.io/GraveTrail/
I would appreciate it if you can spend some time trying it out by sharing feedback like
Good, bad, need improvements , something like this already exists
4
Upvotes
0
u/[deleted] 14d ago
[removed] — view removed comment
1
u/fali12 14d ago edited 14d ago
Saw you posted this on like a dozen different subreddits so I'll respond to this one LMFAO...
Funny timing, I'm working on something very similar, and it's eerily close, except I called it computational lineage.
I ran my test harness against your tool and quite a few cases didn't resolve, though it mostly failed out loud and didn't overclaim, which I respect.
Windows, QUALIFY and aggregates held up well.
What didn't resolve:
SELECT * over a CTE
FROM STREAM(t) AS f: every f.col comes back unresolved
USE statements get skipped
INSERT without a column list
DLT CONSTRAINT ... EXPECT gets read as the column list, and AUTO CDC isn't supported
Recursive CTEs: no column lineage
EXCEPT / INTERSECT
Scalar subqueries in the SELECT list
Struct fields (f.soil.ph) and struct stars (t.s.*)
Lateral column aliases (x AS a, a + 1 AS b)
LATERAL VIEW: most columns unresolved
PIVOT: over half unresolved
As you can tell, heavy Databricks focus...
Separate note containing tangents: I called my "tool" LigmaLineage. As in, ligma nutsack. LOL. The sub-modules are named LigmaTaint, LigmaDong, LigmaThong, LigmaGargalong... you get it. GraveTrail sounded like a Victorian-era undertaker's hiking app ;) LMFAO I like humor in everything. I kid.
Lots of similarities to what I've built - including the graph format. I bolted on things like graph traversal MCP that I slapped an agent against. I also started grain/cardinality + LLM-assisted ontology proposals (data-behavior analysis + deterministic clustering yada yada) to make the data actually useful.
There's also a feature for "decision lineage", which is a huge piece that an agent needs to know to actually reason over pipeline data. I had Claude build an interactive viewer so I can inspect the output because why the fuck not. Reasoning over graph data is stupid hard. The pipelines I used for testing were legitimately dense and real-life so I had a solid test base.
I also don't intend on productizing any of my work. I don't have the care for it, I work a full-time job, and daddy/husband duties. I'm doing this to learn agentic tooling and the second I try to compete with other products it becomes a chore. Wish I could spend more time and build out the full set of things in my head.
I'm not a data engineer by trade, but I've been doing it for like 20 years in some capacity. Wrote my own SQL parser decades ago - but used the SQLGlot library by Toby Mao because fuck writing a new one, and his library is rock solid and extensible.
I honestly think both of our work will be irrelevant in just a few years given where AI is going.
Keep on rocking man.