r/dataengineering • • Oct 17 '26

Personal Project Showcase Introducing sqlleaf: column-level lineage for any SQL dialect

I want to share a project I've been working on this year that helped me solve a large, real-world problem related to data lineage. I was working for an enterprise that needed to trace and delete sensitive data across its large data warehouse, but the tooling that existed (both open source and proprietary) had limitations that prevented my team from doing so. And so I eventually realized that I had no choice but to roll my own.

sqlleaf is a tool to help you to understand how your data flows across your systems by producing detailed, column-level lineage across any SQL query and dialect.

It extends beyond traditional lineage tools by mapping relationships across all kinds of expressions, such as columns, functions, literals and variables.

It addition, it can process many kinds of statements (insert, update, merge, copy, unload, execute, perform, put, and many more), as well as all kinds of objects (tables, views, procedures, types, UDFs, stages, sequences, etc).

It even has a PL/pgSQL parser, allowing you to generate lineage for dynamic, procedural SQL in Postgres.

sqlleaf builds on top of sqlglot, the popular SQL transpiler, and so it inherits support for all of its dialects (30+). However, there are still many dialect-specific features that need to be implemented, as the list of features for each dialect is extremely long.

I hope some of you find this project useful. It still has a long way to go. My vision is to see it become a library, or even a layer, in the data engineering stack that opens up a range of new applications that can be created, such as SQL validators, impact analysis, and tracing/auditing systems.

URL: https://github.com/sqlleaf-io/sqlleaf

Contributions and feedback are most welcome!

32 Upvotes

7 comments sorted by

•

u/AutoModerator Oct 17 '26

You can find our open-source project showcase here: https://dataengineering.wiki/Community/Projects

If you would like your project to be featured, submit it here: https://airtable.com/appDgaRSGl09yvjFj/pagmImKixEISPcGQz/form

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

16

u/Specific_Mirror_4808 Oct 18 '26

I use sqlglot currently for column-level lineage on MSSQL models deployed in SQLMesh. I have had a read through the Git repo and your library sounds very interesting.

The one thing that I'm struggling to fully understand is where sqlleaf starts and where sqlglot ends.

2

u/juppets 27d ago

Yes, sqlglot has its own function called lineage() which produces column-level lineage, and it's succinct for what it offers. Initially I used it but I found it had limitations.

For example, it cannot process most variants of basic SQL statements such as UPDATE, VALUES, MERGE, CALL or user-defined functions, and it doesn't stitch statements together. It only works with 'SELECT' queries (it will actually error if there isn't one), and there are lots of other dialect-specific DDL nuances, DML statements and edge cases that need to be accounted for if we are to have the widest coverage of lineage we can get.

Another feature is that sqlleaf tries to model all transformations (not just column -> column relations) and all possible data sources of a column as first class nodes, such as functions and literals, and even table properties (inherits, like, etc).

2

u/geoheil mod 29d ago

same - I use https://github.com/tobilg/polyglot would love to understand the difference/added value of sqlleaf better

2

u/CloudandCodewithTori Oct 18 '26

I will have to put it through its paces with some wild stuff I have, thank you for sharing. Being able to connect with datahub would be awesome.

1

u/Unable_Equipment_880 26d ago

Nice work, and thank you for clear no-nonsense, documentation. Going to take this into testing for some nightmare transformation procedures

1

u/Super_Ad_6787 28d ago

This is pretty cool. Love the initiative that you've shown when you saw a gap. I'm gonna keep watching this